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Executive  Summary 

Air  Traffic  Control  Specialists  (ATCSs)  work  in  a  dynamic,  visually  challenging  environment 
that  constantly  demands  their  attention.  They  must  monitor,  process  information,  and  make 
decisions  under  conditions  where  taskload  varies  across  a  range  of  their  capabilities.  Engineering 
Research  Psychologists  in  the  National  Airspace  System  Human  Factors  Branch  at  the  Federal 
Aviation  Administration  William  J  Hughes  Technical  Center  used  real  time  person-imthe-loop 
simulation  to  study  these  issues.  They  evaluated  actual  controller  performance  under  two  levels 
of  task  load.  They  also  evaluated  the  impact  of  visual  noise  in  the  form  of  overflights  to  see  if  it 
influenced  workload  and  performance.  This  was  a  concept  research  effort  to  see  if  these 
variables  interacted  to  influence  human  performance  and  controllers’  use  of  the  visual 
information  displayed  for  them. 

Twelve  volunteer  Full  Performance  Level  ATCSs  from  a  Terminal  Radar  Approach  Control 
(TRACON)  facility  participated  in  the  study.  The  ATCSs  worked  simulated  traffic  under 
relatively  low  (6  aircraft  for  each  15  minutes)  and  relatively  high  (12  aircraft  for  each  15 
minutes)  conditions.  Overflights  provided  scenarios  with  the  effect  of  visual  noise  with  two 
levels  of  traffic.  In  addition,  six  scenarios  contained  incursions  into  Class  C  airspace. 

The  results  of  a  study  like  this  are  complex  and  involve  multiple  variables.  Each  variable  has  a 
unique  meaning  in  the  overall  pattern.  Some  findings  can  be  predicted  based  on  past  research 
and  some  could  not.  For  example,  the  over-the-shoulder  observer  estimated  that  controller 
performance  declined  under  conditions  of  higher  task  load.  The  objective  measures  of 
performance  in  fact  showed  that  controller  performance  did  not  decrease. 

The  participating  ATCSs  also  felt  that  they  worked  harder  but  the  quality  of  control  was  lower 
during  the  high  traffic  load  scenarios.  This  is  a  typical  finding  in  simulation  studies  and  could  be 
predicted.  The  self-reported  Situation  Awareness  measures  decreased  under  high  traffic  load. 
Generally,  ATCSs  were  willing  to  indicate  perceived  increases  in  workload,  which  increased 
with  higher  traffic  loads. 

Visual  noise  or  overflights  in  the  TRACON  environment  had  a  complex  impact  on  controller 
perceptions  depending  on  the  task  demand  under  which  they  were  working.  If  they  were  already 
busy  with  traffic  of  their  own,  visual  noise  had  little  impact  and  may  have  even  reduced 
controllers  perceived  workload.  However,  during  slower  times  in  their  own  airspace,  the  fact 
that  they  could  see  that  someone  else  was  using  the  area  that  they  were  scanning  added  to  their 
perceived  workload.  This  suggests  the  advantage  of  filters  at  least  on  an  optional  basis,  where 
appropriate. 

Some  of  the  most  interesting  findings  in  this  study  came  from  the  visual  scanning  data  collected 
with  an  eye  tracker,  referred  to  as  an  oculometer.  This  device  tracks  the  movement  of  the 
controller’s  right  eye  as  it  scans  displays  for  information.  The  system  also  determines  where  on 
the  dynamic  display  the  controller  is  actually  looking.  Visual  scanning  data  included  information 
about  eye  movement  pauses  or  fixations,  eye  jumps  or  saccades,  blinks,  and  pupil  diameter.  The 
human  visual  system  can  only  acquire  detailed  information  during  fixations. 
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Controllers  spent  most  of  their  time  fixating  on  aircraft  targets  and  data  blocks.  Fixation  time 
increased  significantly  when  high  altitude  overflights  were  present.  With  an  increase  in  traffic 
load,  the  number  of  fixations  on  the  radarscbpe  decreased,  but  the  number  of  fixations  on  the 
keyboard  increased.  This  suggests  that  controllers  were  spending  more  time  updating  data  using 
the  keyboard  and  less  time  looking  at  the  radarscope.  The  high  altitude  overflights  seemed  to 
further  divert  the  ATCSs’  attention.  Fixations  on  aircraft  representations  on  the  radarscope 
lasted  longer  than  fixations  on  any  other  item.  These  results  suggest  that  ATCSs  performed  more 
mental  processing  when  looking  at  the  radarscope  and  aircraft  representations  in  particular  than 
when  looking  at  any  other  object.  Controllers  developed  patterns  of  visually  scanning  the  radar 
display.  These  patterns  became  more  stractured  as  the  traffic  situation  developed.  ATCSs  did 
not  change  these  patterns  with  the  advent  of  aircraft  intrusions  into  the  airspace.  This  may 
explain  in  part  why  they  noticed  these  unscheduled  targets  late  or  not  at  all.  In  the  interests  of 
airspace  safety,  it  is  not  enough  to  display  intrusive  targets.  Their  presence  must  be  emphasized 
in  a  way  to  draw  the  controller’s  attention  away  from  his/her  established  scanning  pattern  so  that 
he/she  can  amend  plans  and  avoid  potential  conflicts. 

This  research  provides  greater  understanding  of  how  ATCSs  use  current  information  displays. 
The  research  method  has  potential  for  increasing  future  ATCS  efficiency  through  improved 
display  technology  or  new  training  techniques. 


X 


1.  Introduction 


The  Federal  Aviation  Administration  (FAA)  started  a  controller  information  scanning  program  in 
1989  to  help  understand  and  reduce  errors  (Stein,  1989).  With  applications  to  Air  Traffic 
Control  (ATC)  training,  error  analysis,  and  equipment  design  evaluation,  the  identification  of  An- 
Traffic  Control  Specialists’  (ATCSs’)  visual  scanning  patterns  and  quantification  of  these 
patterns  are  necessary.  Presently,  no  objective  measures  of  visual  scanning  exist  to  support  this 
program. 

This  was  the  second  in  a  series  of  visual  scanning  studies  of  ATCSs  conducted  at  the  FAA 
William  J.  Hughes  Technical  Center  Research  Development  &  Human  Factors  Laboratory 
(RDHFL)  at  Atlantic  City  International  Airport,  New  Jersey.  The  first  study  (Stein,  1992) 
addressed  the  effect  of  changes  in  traffic  density  on  visual  scanning.  With  the  technology  at  the 
time,  the  experimenters  could  not  synchronize  the  visual  scanning  patterns  with  air  traffic  events. 
This  RDHFL  study  was  the  first  to  use  head-mounted  oculometry  synchronized  with  a  dynamic 
Air  Traffic  Simulator. 

This  exploratory  project  forms  the  basis  for  analyses  on  Visual,  performance,  and  questionnaire 
data.  The  project  compared  behavior  and  performance  of  ATCSs  across  experimental 
conditions. 

1.1  Background 

In  1995,  the  National  Aeronautics  and  Space  Administration  (NASA),  the  Department  of 
Defense,  and  the  FAA  published  the  National  Plan  for  Civil  Aviation  Human  Factors.  The 
purpose  of  this  plan  was  to  enhance  aviation  safety  and  improve  the  efficiency  of  operations.  It 
identified  research  areas  and  emphasized  tire  transfer  of  research  findings  to  planned  and  ongoing 
programs.  One  of  the  key  issues  of  the  plan  was  to  quantity  the  effect  that  new  products  or 
procedures  have  on  system  and  human  performance.  The  plan  consists  of  five  areas:  Human- 
Centered  Automation,  Selection  and  Training,  Human  Performance  Assessment,  Information  and 
Management  and  Display,  and  Bioaeronautics.  Except  for  Bioaeronautics,  each  of  these  areas 
states  specific  research  areas  that  require  a  national  focus.  Visual  scanning  related  measures 
have  a  potential  application  across  most  of  these  research  areas. 

The  duties  of  an  ATCS  involve  scanning,  projecting,  planning,  and  execution.  A  radar  display 
and  flight  progress  strips  provide  visual  data,  whereas  radio  and  telephone  communication 
systems  provide  auditory  data.  The  cognitive  requirements  of  ATC  involve  the  processing  of 
dynamically  changing  information  (Kirchner  &  Laurig,  1971 ;  Means  et  al.,  1988).  The  ATCS 
develops  an  underlying  mental  model  of  the  ATC  situation.  This  model  allows  the  ATCS  to 
switch  attention  between  the  various  data  sources  (Guttman,  Stein,  &  Gromelski,  1995;  Mogford, 
Muiphy,  Roske-Hofstrand,  Yastrop,  &  Guttman,  1994;).  In  this  study,  human  factor  specialists 
conducted  simulations  in  real  time  and  collected  data  on  visual  scanning,  performance,  and 
mental  woridoad. 

Researchers  have  used  workload  and  performance  measures  extensively  to  test  design 
alternatives  in  the  ATC  environment.  In  an  early  visual  attention  study,  Karsten,  Goldberg, 
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Rood,  and  Sultzer  (1975)  found  that  ATCSs  spend  approximately  80%  of  their  time  looking  at 
the  radar  display,  13%  looking  at  flight  strips,  and  5%  looking  at  input  devices.  Their  equipment 
was  primitive  by  current  standards.  With  the  advancement  of  technology  and  recent 
enhancements  in  software  and  hardware,  the  RDHFL  now  simulates  the  ATC  environment  with  a 
much  higher  degree  of  fidelity. 

1.1.1  Literature  Related  to  Visual  Scannin  g 

The  amount  of  sensory  information  available  to  a  human  being  at  any  one  point  in  time  is 
1,000,000,000  bits  per  second  at  the  human  sensory  level  (Grandjean,  1993).  This  information, 
although  highly  filtered  before  reaching  conscious  awareness,  is  still  of  critical  importance  to  the 
perform^ce  of  everyday  activities.  The  most  relied  upon  sensory  information  comes  fi-om  the 
visual  system  having  approximately  90%  of  a  person’s  daily  activities  under  its  guidance. 

The  visual  system  provides  information  about  the  ATC  environment  necessary  to  anticipate 
changes  and  to  react  appropriately.  When  looking  at  an  object,  the  eyes  move  rapidly  from  one 
point  of  interest  to  another.  These  fast  jumps,  called  saccades,  are  ballistic  movements  fiiat,  once 
started,  will  continue  rmtil  they  reach  their  target  destination  (Carpenter,  1977).  During  a 
saccade,  the  visual  system  obtains  little  visual  information  other  than  the  detection  of  movement. 
Most  of  the  time,  humans  look  at  objects  without  moving  their  eyes.  During  these  stationary 
periods  between  saccades,  called  fixations,  humans  register  most  visual  information.  In  a  30- 
minute  scenario,  ATCSs  have  roughly  3600  fixations  with  an  average  duration  of  approximately 
500  ms  (Stein,  1992). 

A  fixation  is  a  four-part  process.  First,  the  visual  system  stores  an  image  in  short-term  visual 
memory.  Second,  the  visual  system  encodes  the  raw  image  and  stores  the  codes  in  working 
memory.  In  the  third  stage,  further  mental  processing  takes  place  and,  in  the  fourth  stage,  the 
visual  system  prepares  for  the  next  saccade.  The  preparation  time  for  the  next  saccade  increases 
with  an  increase  in  the  magnitude  of  the  future  saccade  (Kapoula,  1983).  Kapoula  showed  that 
the  proximity  of  previous  fixations  influenced  fixation  duration  on  subsequent  points  of  interest. 

Like  most  human  neuromotor  control  systems,  the  oculomotor  system  uses  open  and  closed  loop 
control,  depending  on  the  situation.  In  closed  loop  control,  information  acquired  during  a 
fixation  directs  the  subsequent  saccade  (Kapoula,  1983;  Rayner  &  Pollatsek,  1981;  Vaughn, 

1982).  The  visual  system  uses  closed-loop  control  in  active  information  searching  during 
situations  with  potential  points  of  interest  in  close  proximity,  hi  open-loop  control,  information 
processing  independent  of  the  current  visual  information  in  the  visual  field  determines  the  next 
saccade  (Ellis,  1986).  An  open-loop  system  scans  the  visual  field  in  the  periphery  for  potential 
points  of  interest.  Higher  level  cognitive  processes  determine  the  target  of  the  next  saccade  in 
open-loop  control. 

Experienced  participants  tend  to  scan  for  pertinent  information  in  a  stratified  random  manner 
(Card,  1983;  Engle,  1977;  Groner  &  Groner,  1982;  Inditsky  &  Bodmann,  1980;  Kraiss  & 
Knauper,  1983;  Krendel  &  Wodinsky,  1960;  Senders,  1966;  Weir  &  Klein,  1970;  Wewerinke, 
1981).  A  structured  model  gives  priority  to  objects  or  groups  that  need  more  attention  while 
updating  the  total  picture  of  the  process  under  control.  Less  experienced  participants  do  not  have 
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a  well-structured  model  available  in  long  term  memory  and  tend  to  follow  events  that  can  lead 
diem  astray.  An  example  is  tunneling,  when  an  ATCS  loses  the  overall  picture  and  focuses  on  a 
single  problem  only. 

1.1.2  Literature  Related  to  W orkload 

Studies  aimed  at  improving  the  safety  of  air  traffic  often  include  ATCSs’  performance  and 
workload.  Researchers  have  developed  a  variety  of  assessment  techniques  to  evaluate  workload. 
Subjective  techniques  have  dominated  this  research  area  because  of  the  ease  of  administration, 
low  cost,  and  lack  of  obtrusiveness.  The  variety  of  available  measures  indicates  a  lack  of 
consensus  among  researchers  and  presents  an  obstacle  when  attempting  to  generalize  and 
integrate  research  findings.  The  NASA  Task  Load  Index  (TLX)  (Hart  &  Staveland,  1988)  and 
the  Subjective  Workload  Assessment  Technique  (SWAT)  (Reid  &  Nygren,  1988)  serve  a  wide 
variety  of  research  needs.  The  TLX  and  the  SWAT  assess  mental  workload  at  the  end  of  the 
scenario  or  experiment  and  break  down  mental  workload  into  several  components.  Other 
subjective  mental  workload  assessment  techniques  follow  a  more  holistic  approach.  The  An- 
Traffic  Workload  Input  Technique  (ATWIT)  (Stein,  1985),  derived  from  earlier  work  by  Stein 
and  Rosenberg  (1983),  uses  a  single  10-point  scale  to  assess  perceived  workload.  The  ATWIT 
collects  assessments  of  perceived  workload  during  the  scenario.  An  experiment  should 
incorporate  both  objective  and  subjective  measures  to  fully  assess  workload. 

When  reaching  working  memory  limits,  mental  workload  increases  and  performance  decreases. 
Performance  shows  an  inverted  U-shaped  dependency  on  workload  with  poor  performance 
occurring  at  extremely  low  and  high  mental  workload  levels.  Optimal  performance  will  often 
occur  between  these  two  extremes  (Tole,  Stephens,  Hams,  &  Ephrath,  1982). 

1.2  Purpose 

The  study  explored  the  eye  movement  characteristics  of  Terminal  Radar  Approach  Control 
(TRACON)  Full  Performance  Level  (FPL)  ATCSs  under  different  levels  of  task  load,  with  and 
without  overflying  aircraft  (visual  noise),  and  with  and  without  aircraft  intrusions.  It  answered 
seven  research  questions  that  addressed  visual  scanning,  subjective  ratings,  over-the-shoulder 
(OTS)  ratings,  questionnaire  scores,  and  performance  scores. 

Depending  on  the  scoiario,  the  ATCS  encountered  airspace  intrusions,  different  task  loads,  and 
en  route  aircraft  primary  radar  returns.  Researchers  determined  if  changes  in  experimental 
conditions  altered  performance  and  behavior.  The  questions  related  to  these  changes  are  as 
follows: 

a.  Do  eye  movement  characteristics  of  ATCSs  differ  across  experimental  conditions? 

b.  Do  subjective  mental  workload  estimates  (ATWIT)  differ  across  experimental 
conditions? 

c.  Do  OTS  ratings  differ  across  scenarios? 

d.  Do  responses  to  Post-Scenario  Questionnaires  differ  across  scenarios? 

e.  Do  performance  scores  differ  across  expaimental  conditions? 
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f.  Do  eye  movement  characteristics  differ  depending  on  Visual  Flight  Rules  (VFR) 
intrusion  presence? 

g.  Do  eye  movement  characteristics  differ  depending  on  Instrument  Flight  Rules  (IFR) 
intrusion  presence? 

1.3  Scope 

This  study  compared  visual  scanning  behavior,  system  activity,  ATCS  performance,  workload, 
and  pilot-ATCS  interactions  under  conditions  that  differed  in  traffic  load,  presence  of  visual 
noise,  and  aircraft  intrusion  in  Class  C  terminal  airspace. 

2.  Method 

2.1  Participants 

Twelve  active  FPL  ATCSs  from  a  TRACON  facility  participated  in  the  study.  The  participants 
actively  controlled  traffic  for  at  least  16  hours  in  the  month  preceding  the  experiment.  The 
ATCSs  gave  their  verbal  informed  consent  to  participate  in  the  experiment.  The  research  team 
ensured  them  that  their  data  were  completely  confidential.  Participants  had  visual  acuity  not  less 
than  20/30  corrected.  ATCSs  could  wear  corrective  lenses  and  soft  contact  lenses.  The 
oculometer  design  limitations  excluded  bifocals,  trifocals,  or  hard  contact  lenses. 

2.2  Facility 

The  experiment  took  place  in  three  areas  of  the  RDHFL:  Experiment  Room  Four  (ER4), 
Experiment  Observation  Room  Two  (EOS2),  and  Experiment  Room  2  (ER2).  ER4  contains  a 
hi^  fidelity,  state-of-the-art  ATC  simulator  run  by  ATCoach  (1992)  simulation  software.  This 
station  can  mimic  up  to  an  ARTS  IDA  radar  system  and  consists  of  a  22-inch,  high-resolution 
(2000  X  2000  pixels)  color  radar  display,  a  three-button  trackball,  and  an  ARTS  IHA  keyboard. 
The  system  operated  in  networked  mode  linked  to  the  ER2  that  contained  the  simulation  pilot 
workstations.  ER4  and  EOS2  contained  video  cameras  and  recorders  synchronized  with 
ATCoach,  the  ATWIT  panels,  and  UNIX  network  hardware.  The  simulation  workstation 
included  a  flight  strip  bay  with  time-ordered  fli^t  progress  strips.  The  staff  modeled  the 
TRACON  and  interfaced  ATCoach  with  an  Applied  Science  Laboratories  (ASL)  oculometer. 
The  oculometer  consists  of  an  eye/head  tracking  system  that  recorded  the  point  of  gaze  (POG) 
and  pupil  diameter  of  a  person  by  using  near  infixed  reflection  outlines  from  the  pupil  and 
cornea.  For  a  detailed  description  of  the  equipment  used  in  the  simulation,  see  Appendix  A. 

2.2.1  Support  Personnel 

The  study  employed  three  simulation  pilots.  To  allow  rotation,  researchers  trained  nine 
simulation  pilots  using  procedures  from  past  experiments  with  additional  procedures  for  VFR 
aircraft.  One  simulation  pilot  read  back  clearances.  A  second  simulation  pilot  keyed  in  entries 
sent  to  the  computer  that  updated  the  movement  of  the  displayed  aircraft.  The  third  simulation 
pilot  manually  recorded  simulation  commands  corresponding  to  clearances.  The  training  of  the 
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simulation  pilots  lasted  3  weeks.  Training  included  procedures  related  to  simulation  pilots’ 
commands  and  familiarization  of  simulation  equipment.  The  simulation  pilots  trained  at  every 
position. 

A  research  team  composed  of  a  research  psychologist,  a  human  factors  engineer,  and  a  subject 
matter  expert  (SME)  conducted  the  simulations.  The  team  created  the  scenarios,  conducted  the 
OTS  ratings  and  the  experiments,  performed  the  data  analyses,  and  wrote  the  final  technical 
report.  RDHFL  support  engineers  ensured  that  die  hardware  and  software  functioned  properly. 

2.3  Operation 

During  the  simulations,  a  personal  computer  recorded  the  eye  movements.  The  simulator 
software  recorded  aircraft;  activities.  Off-line  software  programs  integrated  the  POG  data  and  the 
data  provided  by  the  simulator.  Programs  developed  by  RDHFL  software  engineers  reduced  the 
eye  movement  data  and  calculated  fixation,  saccade,  blink,  and  pupil  characteristics.  For  each 
fixation,  the  software  determined  the  radarscope  objects  (aircraft,  airports,  fixes,  etc.)  within  a  2- 
inch  radius  fi:om  the  center  of  a  fixation. 

2.4  Design 

The  objective  of  this  study  was  to  compare  visual  scan  patterns  of  ATCSs  during  high  and  low 
task  load,  presence  and  absence  of  visual  noise,  and  presence  and  absence  of  VFR  or  IFR 
intrusionL  The  design  was  a  2  x  2  (task  load  x  overflight)  repeated  measures  full  factorial 
design.  Task  load  had  two  levels,  low  (6  aircraft  per  15  minutes)  and  high  (12  aircraft  per  15 
minutes),  and  there  were  scenarios  with  and  without  overflights. 

2.4.1  Independent  Variables 

The  independent  variables  (IVs)  were  visual  noise,  task  load,  and  intrusions.  Visua.1  noise  arid 
task  load  differed  between  scenarios,  whereas  intrusion  type  changed  within  scenarios  over  time. 
Each  scenario  consisted  of  simulated  air  traffic  of  the  TRACON  modeled  in  ATCoach  for 
previous  experiments  (Guttman  et  al.,  1995) 

The  experiment  included  scenarios  with  and  without  visual  noise.  In  the  visual  noise  condition, 
researchers  modeled  overflying  aircraft  into  the  scenario  as  visual  noise  using  primary  radar 
returns.  In  the  no  visual  noise  condition,  there  were  no  overflights.  Flight  strips  from  an  Air 
Route  Traffic  Control  Center  (ARTCC)  formed  the  basis  for  the  calculation  of  the  number  of 
aircraft  and  the  traffic  composition  of  all  overflights. 

The  research  team  varied  traffic  volume  and  traffic  fi-equency  across  scen^os.  The  low  task 
load  condition  had  an  average  of  6  aircraft  entering  the  airspace  per  15  ^utes  with  6  aircraft 
visible  on  the  radar  screen  at  any  given  time.  The  high  task  load  condition  had  an  average  of  12 
aircraft  per  15  minutes  with  12  aircraft  visible  on  the  radar  screen  at  any  given  time.  The  actual 
scenario  composition  varied  depending  on  how  the  ATCS  worked  the  airspace. 
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The  simulations  included  intrasions  as  aircraft  making  an  unscheduled  entry  into  Class  C 
airspace.  The  intrusions  included  botii  aircraft  under  VFR  or  IFR  with  special  care  given  to 
prevent  the  ATCS  from  anticipating  the  onset  of  an  intrusion.  The  levels  of  the  intrusion  IVs 
were  no  intrusion  (baseline),  VFR  intrusion,  or  IFR  intrusion. 

The  research  team  created  eight  scenarios  reflecting  the  levels  of  the  IVs  [overflights  (yes,  no), 
task  load  (low,  high),  and  intrusion  type  (FR,  VFR)].  For  a  detailed  description  of  the 
experimental  and  practice  scenarios,  see  Appendix  B.  The  TRACON  used  in  these  scenarios 
consisted  of  two  sectors  (north  and  south),  worked  by  a  single  ATCS.  To  keep  the  scenarios 
realistic,  they,  at  most,  included  two  intrusions.  FR  intrusions  only  occurred  under  the 
overflight  condition. 

2.4.2  Dependent  Variables 

Researchers  averaged  the  following  sets  of  dependent  variables  (DVs)  over  5-minute  intervals: 

a.  Subjective  Workload  Assessment.  The  ATWIT  device  (Stein,  1985)  assessed  the 
workload  of  the  ATCS.  The  ATWIT  measure  is  a  workload  estimate  based  on  a  scale 
from  1  (very  low  or  no  workload)  to  1 0  (extremely  high  workload).  The  ATCS, 
prompted  by  a  low  tone,  made  a  workload  rating  every  5  minutes.  Each  participant  made 
9  ATWIT  ratings  in  a  45-minute  scenario  allowing  calculation  of  the  mean  and  maxiTmim 
rating  for  each  scenario. 

b.  Questionnaires.  The  experimenters  used  three  types  of  self-report  questionnaires  adapted 
from  previous  experiments.  The  questionnaires  (see  Appendix  C)  included  an  Entry 
Questionnaire,  Post-Scenario  Questionnaire,  and  Exit  Questionnaire  (Abbott,  Nataupsky, 
&  Steinmetz,  1987;  Guttman  et  al.,  1995;  Sollenberger  &  Stein,  1995;  Stein,  1992).  The 
Entry  Questionnaire  contained  questions  concerning  demographic  information.  The  Post- 
Scenario  Questionnaire  contained  questions  about  various  aspects  of  controlling  trafflc 
during  a  scenario.  The  Exit  Questionnaire  provided  feedback  about  the  experiment. 

c.  Over-the-Shoulder  Ratings.  The  research  team  rated  the  performance  of  the  ATCSs  for 
each  scenario.  They  used  a  form  that  captures  a  wide  range  of  ATC-related  performance 
issues  (adapted  from  Guttman  et  al.,  1995).  (See  Appendix  D.) 

d.  Performance.  The  automated  data  reduction  module  developed  at  the  RDHFL  provided 
performance  data  broken  down  by  conflicts,  complexity,  error,  communications,  and  task 
load  (Algeo  and  Pomykacz  (1996).  Further  analysis  used  a  subset  of  these  performance 
variables  (see  Appendix  E). 

e.  Visual  Scanning.  The  oculometer  data  formed  the  basis  for  the  variables  related  to  visual 
scanning.  For  each  scenario  and  5-minute  interval,  the  research  team  calculated  the 
variables  in  Appendix  F,  Table  F-2.  Visual  scanning  targets  were  radarscope,  keyboard 
area,  ATWIT  device,  flight  strip  bay,  aircraft,  static  objects,  departure  list,  system 
settings,  preview  area,  and  Conflict  Alert/Low  Altitude  (CA/LA)  area.  See  Appendix  F 
for  a  more  detailed  description  and  information  about  the  computation  of  the  visual 
scanning  DVs. 
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2.5  Procedure 


Twelve  FPL  ATCSs  participated  in  the  experiment  diiring  the  workweek.  The  morning  of  then- 
first  day  consisted  of  a  briefing  and  a  familiarization  period.  The  research  team  explained  the 
experiment,  the  oculometer,  differences  between  ATCoach  and  their  own  equipment,  and  the 
confidentiality  of  ATCSs’  identity.  They  provided  an  informed  consent  briefing,  and  participants 
gave  a  verbal  commitment  to  the  experiment  and  their  understanding  of  informed  consent 
doctrine.  The  ATCSs  then  completed  an  Entry  Questionnaire  that  included  demographic 
questions  about  age,  experience  level,  and  need  for  corrective  glasses.  Researchers  assigned  the 
participants  to  an  experimental  condition. 

After  receiving  instructions  about  the  Letter  of  Agreement  (LOA)  and  the  Standard  Operating 
Procedures  (SOPs),  the  ATCSs  familiarized  themselves  with  the  laboratory  equipment.  The 
laboratory  equipment  included  the  2K  display  and  the  simulation  configuration  of  the  sector. 
Then,  the  ATCSs  completed  a  20-minute  familiarization  scenario  with  the  oculometer.  After  a 
break,  the  first  of  three  scenarios  was  run.  Each  experimental  run  consisted  of  setup  and 
calibration  of  the  oculometer,  a  simulation  run,  and  a  Post-Scenario  Questionnaire.  After  the 
initial  scenario,  there  was  a  break  for  lunch  after  which  the  ATCSs  worked  two  scenarios  with  a 
30-minute  break  between  each  scenario.  The  second  day  consisted  of  a  brief  simulation  review 
followed  by  two  scenarios  in  the  morning  and  three  scenarios  in  the  afternoon.  Finally,  the 
participants  filled  out  an  Exit  Questionnaire.  Appendix  G  presents  a  detailed  schedule  of 
activities. 

2.5.1  Data  Reduction 

2.5. 1.1  Questionnaires 

Researchers  administered  the  Entry,  Post-Scenario,  and  Exit  Questionnaires  in  paper  and  pencil 
format  and  transcribed  the  responses  into  a  spreadsheet.  Researchers  created  a  data  set  for  each 
questionnaire. 

2.5.1 .2  Over-the-Shoulder  Ratings 

Researchers  entered  the  ratings  from  the  OTS  questionnaires  into  a  spreadsheet.  The  data  set 
consisted  of  SME  ratings  of  each  ATCS  for  all  eight  scenarios. 

2.5.1 .3  Visual  Scanning  Data 

The  oculometer  recorded  eye  movements  in  terms  of  horizontal  and  vertical  positions.  The 
Magnetic  Head  Tracker  (MHT)  provided  position  and  orientation  of  the  head  in  six  degrees  of 

freedom.  The  software  integrated  the  eye  and  head  movement  data  to  determine  the  POG.  The 

oculometer  identifies  the  plane  at  which  the  ATCS  looked  and  records  the  coordinates  relative  to 
that  plane.  The  sampling  rate  of  the  oculometer  and  the  MHT  was  60  samples  per  second. 
Experimenters  reduced  the  raw  data  and  expressed  it  as  fixations,  saccades,  and  blinks.  Fixation 
characteristics  included  time  of  onset,  duration,  the  plane  being  looked  at,  the  area  covered  by 
small  eye  movements  within  the  fixations,  and  the  coordinates  relative  to  the  plane.  Appendix  H 
contains  a  description  of  the  output  after  this  first  stage  of  data  reduction.  Saccade  characteristics 
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include  information  on  the  magnitude  of  the  saccade  and  the  average  velocity  during  the  saccade. 
Researchers  summarized  a  number  of  variables  derived  from  the  fixation  and  saccade  data  per 
scenario  and  5-minute  interval.  The  first  data  set  contained  8x12  (scenarios  x  ATCSs)  records 
of  the  visual  scanning  summary  variables  per  scenario.  The  records  contained  ATCS  and 
experimental  condition  identifications  at  the  scenario  level.  The  second  set  contained  8x12x9 
(scenarios  x  ATCSs  x  intervals)  records  of  the  visual  scanning  summary  variables  per  5-minute 
interval. 

The  research  team  integrated  the  eye  movement  data  with  simulator  information  about  static 
objects  (airports,  VHP  Omni-directional  ranges  (VORs),  fixes,  intersections,  and  the  system  area) 
and  dynamic  objects  (aircraft  and  the  preview  area).  Appendix  I,  Figure  I-l  displays  a  snapshot 
at  20  minutes  into  a  high  task  load  scenario  with  visual  noise  present.  Appendix  I,  Figure  1-2, 
presents  the  integrated  data  of  the  simulator  and  the  oculometer  for  a  similar  scenario. 

Figures  1-3  and  1-4  show  the  advantage  of  collecting  object-related  fixation  information.  Figure 
1-3  shows  the  fixations  of  one  participant  for  a  45-niinute  low  task  load  scenario  without  visual 
noise.  Although  one  sees  an  increased  density  of  the  number  of  fixations  along  the  runways 
(shown  in  Figure  1-3),  no  information  is  available  about  how  this  relates  to  the  fixation 
distribution  across  aircraft.  Superimposing  the  flight  paths  of  the  20  aircraft  in  the  scenario  did 
not  relate  fixation  information  to  aircraft  movements.  Identifying  a  target  aircraft  (e.g., 
BTA3721)  clearly  shows  that  the  ATCS  follows  that  aircraft  throughout  the  airspace  (Figure  1-4). 

2.5.1 .4  Performance  Variables  and  ATWIT 


The  Data  Reduction  &  Analysis  (DR&A)  module  processed  raw  data  files  produced  by 
ATCoach,  ATWIT,  and  the  communications  system.  The  DR&A  module  produced  summary, 
interval,  and  error  files  for  each  scenario.  The  interval  and  summary  files  formed  two  separate 
data  sets.  The  first  data  set  contained  12x8  (ATCSs  x  scenarios)  records  that  included  the 
summary  variables  calculated  per  scenario.  The  second  data  set  with  12x8x9  (ATCSs  x 
scenarios  x  intervals)  records  contained  the  summary  variables  calculated  per  5-inmute  interval. 

2.5.2  Data  Analysis 

This  section  briefly  describes  the  data  analysis  for  DV  data  sets  (ATWIT,  questionnaires,  OTS 
rating  form,  visual  scanning,  and  performance).  The  statistical  methods  used  for  the  analysis 
include  Multivariate  Analysis  of  Variance  (MANOVA)  and  Analysis  of  Variance  (ANOVA). 

The  MANOVA  compares  averages  for  several  variables  simultaneously,  tests  if  these  averages 
are  different  due  to  chance  alone,  and  includes  the  effects  of  more  than  one  DV.  After  a 
significant  result  of  a  MANOVA,  researchers  conducted  ANOVAs  to  investigate  individual  DVs. 

The  ANOVAs  compare  averages  of  a  single  variable  between  multiple  conditions  and  determines 
if  these  averages  are  different  due  to  chance  alone.  A  difference  between  means  is  significant  if 
there  is  a  very  high  probability  that  the  means  are  actually  different.  For  general  concepts  in 
statistics  and  more  detailed  information  about  the  statistical  methods  used  in  this  study,  see 
Appendix  J. 
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2,5.2. 1  ATWrr  Ratings 


For  the  analysis  concerning  the  subjective  ratings,  researchers  used  a  MANOVA  on  maximum 
and  mean  ATWIT  ratings.  This  MANOVA,  structured  as  a  2  x  2  (Task  load  x  Visual  noise) 
rq)eated  measures  design,  addressed  the  differences  across  scenarios. 

2.5.2.2  Questionnaires 

The  Entry  Questionnaire  contained  questions  about  participant  background  and  importance  of 
provided  airspace  and  aircraft  information.  The  analysis  of  the  Entry  Questionnaire  data 
consisted  of  the  calculation  of  means  and  standard  deviations  (SD). 

The  Post-Scenario  Questionnaire  contained  general  questions  about  the  simulation,  ATCSs’ 
perceived  Situation  Awareness  (SA),  and  NASA  TLX  items.  If  the  MANOVA  showed  statistical 
significance,  subsequent  analyses  included  ANOVAs  on  the  individual  variables.  The  analyses 
of  the  SA  and  NASA  TLX  items  followed  the  same  pattern  as  the  analyses  of  Ihe  general 
questions. 

The  Exit  Questionnaire  collected  ATCSs’  impressions  of  the  experiment.  The  analysis  of  the 
Exit  Questionnaire  data  consisted  of  the  calculation  of  means  and  SDs. 

2.5.2.3  Over-the-Shoulder  Ratings 

The  OTS  ratings  consist  of  questions  relating  to  six  categories:  Maintaining  Safe  and  Efficient 
Traffic  Flow,  Maintaining  Attention  and  SA,  Prioritizing,  Providing  Control  Information, 
Technical  Knowledge,  and  Communication.  The  researchers  compared  OTS  rater  responses  in  a 
two-way,  2x2  (overflights  x  task  load)  fashion. 

2.5.2A  Visual  Scanning 

Three  MANOVAs  tested  the  hypotheses  related  to  the  changes  in  visual  scanning.  The  first 
MANOVA  addressed  visual  scanning  differences  across  scenarios  and  was  a  2  x  2  repeated 
measures  analysis  (overflights  x  task  load).  The  second  MANOVA  addressed  the  differences 
between  5-minute  intervals  in  similar  scenarios  that  contained  VFR  intrusions  and  the 
corresponding  interval  without  intrusions.  It  was  a  two-way  repeated  measures  MANOVA  (i.e., 
2x5  [VFR  presence  x  conditions]).  The  third  MANOVA  investigated  differences  between 
intervals  in  similar  scenarios  that  contained  IFR  intrusions  and  the  corresponding  intervals 
without  intrusions.  This  MANOVA  was  of  a  2  x  5  (IFR  presence  x  conditions)  design. 

2.5.2.5  Performance  Scores 

The  four  categories  of  variables  related  to  performance  included  conflicts,  separation, 
complexity,  and  commumcations.  Four  sets  of  MANOVAs  tested  the  hj^otheses  related  to 
performance  scores  on  selected  performance  variables.  These  MANOVAs  addressed  the 
differences  across  scenarios  and  were  of  repeated  measures  2x2  (overflights  x  task  load)  design. 
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3.  Results 


Analyses  of  the  Entry  and  Exit  Questionnaires  consisted  of  the  calculation  of  the  means  and  SDs. 
Analyses  of  other  data  sets  involved  MANOVAs  and  ANOVAs  when  appropriate.  Appendix  K 
presents  overall  averages  for  DVs  used  in  inferential  statistics. 

3.1  ATWIT 


The  ATWrr  device  recorded  ATCS  ratings  and  the  amoimt  of  time  it  took  the  ATCS  to  respond 
(latencies).  Researchers  calculated  the  mean  and  maximum  ATWIT  rating  and  latency  for  each 
scenario.  Correlations  between  the  mean  and  maximum  on-line  ATWIT  ratings  and  the  post¬ 
scenario  TLX  workload  indicated  what  drives  the  post-scenario  perception  of  workload.  This 
report  only  presents  the  results  of  the  analyses  on  mean  and  maximum  ATWIT  ratings  (Figure  1). 
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ar[i] :  ATWIT  Rating  for  interval  [i] 
al[i] :  ATWIT  Latency  for  interval  [i] 


Figure  1.  Derivation  of  ATWIT  variables  from  raw  ATWIT  scores. 

The  MANOVA  of  the  ATWIT  ratings  included  the  mean  and  the  maximum  of  the  ratings  within 
a  scenario.  The  effects  of  increasing  task  load  and  introducing  visual  noise  interacted  [A  =  .70, 
F(2, 21)  =  4.45, p  <  .05]  (Appendix  L,  Table  L-9).  The  effect  of  visual  noise  was  not  significant 
as  a  simple  effect  (Table  L-9). 

Researchers  included  both  the  mean  and  the  maximum  ATWIT  rating  items  in  the  MANOVA. 

To  ensure  an  overall  alpha  level  of  .05,  the  adjusted  alpha  was  .025  for  the  ANOVAs. 

3.1.1  Mean  ATWIT  Rating 

Under  high  task  load  conditions,  the  mean  ATWIT  rating  was  significantly  higher  than  under  low 
task  load  conditions  [F(l,  22)  =  92.37,  p  <  .05]  (Figure  2),  The  presence  of  visual  noise  did  not 
significantly  affect  the  mean  ATWIT  ratings. 
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Figure  2.  Means  and  SDs  of  mean  ATWIT  ratings  as  a  ftinction  of  task  load. 

3.1.2  Maximum  ATWIT  Rating 

The  effects  of  introducing  visual  noise  and  increasing  task  load  on  the  maximum  ATWIT  rating 
interacted  [F(l,  22)  =  9. 19,^3  <  .05]  (Appendix  L,  Table  L-10).  The  simple  effects  showed  that 
the  effect  of  task  load  on  the  maximum  ATWIT  rating  was  stronger  under  the  no  noise  condition 
(Table  L-1 1).  There  was  no  significant  effect  of  the  presence  of  visual  noise  on  the  maximum 
ATWIT  rating  for  both  task  load  levels  (Figure  3). 


□  Low  Task  Load 
■  High  Task  Load 


No  Visual  Noise  Visual  Noise 

Figure  3.  Means  and  SDs  of  maximum  ATWIT  ratings  for  load-visual  noise  combinations. 

3.1.3  Correlation  Between  Mean  and  Maximum  ATWIT  Ratings  and 

The  post-scenario  TLX  items  showed  higher  correlations  with  the  mean  ATWTT  ratings  than 
with  the  maximum  ATWIT  ratings.  Both  the  mean  and  maximum  ATWTT  rating  showed  the 
highest  correlation  with  the  TLX  item  on  mental  demand  (r  =  .71  and  r  =  .50,  respectively). 
Table  K-3,  Appendix  K,  presents  a  detailed  correlation  matrix. 

3.2  Questionnaires 

3.2.1  Entry  Questionnaire 

The  Entry  Questionnaire  inquired  about  participants’  general  background  and  preferences  of 
information  available  on  aircraft  and  radarscope.  When  asked  to  indicate  an  LOA  or  level  of  a 
modality,  participants  chose  from  a  discrete  1 0-point  scale. 

The  12  participants  averaged  37  years  of  age,  almost  12  years  of  ATC  experience,  and  over  8 
years  at  their  TRACON.  One  third  of  the  participants  used  corrective  lenses  during  the 


Low  Task  Load  High  Task  Load 
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experiments.  These  volunteers  actively  controlled  traffic  for  an  average  of  1 1 .5  months  during 
the  last  12  months.  Their  self-rated  ATC  skill  level  was  high,  and  they  perceived  a  moderate 
stress  level.  Their  motivation  and  current  state  of  health  were  good.  They  indicated  moderate 
preference  towards  vertical  separation,  less  preference  towards  vectoring,  and  no  level  of 
preference  towards  speed  control.  The  self-rated  level  of  experience  with  video  games  was  low. 
Table  1  presents  detailed  values  for  the  means  and  SDs  for  the  general  background  variables. 


Table  1.  General  Background  Questions  (N  =  12) 


Variable  Label 

Mean 

SDs 

37.42 

3.55 

Lenses 

0.33 

ATC  Experience 

11.67 

4.38 

Present  TRACON  Experience 

8.42 

4.62 

Active  Control  last  12  Months 

11.50 

1.73 

ATC  Skill 

8.25 

1.22 

Stress 

5.50 

2.15 

Motivation 

7.42 

2.11 

Health 

8.58 

1.16 

Vertical  Separation  Preference 

6.75 

1.36 

Vectoring  Separation  Preference 

5.67 

1.30 

Speed  Separation  Preference 

4.83 

1.64 

Video  Game  Experience 

3.42 

2.15 

Table  2  presents  die  ratings  for  several  aircraft-related  variables  sorted  from  most  important  to 
least  important.  The  ATCSs  rated  the  current  altitude,  current  location,  and  assigned  altitude  as 
the  three  most  important  pieces  of  information  about  the  aircraft.  Least  important  were  entry  fix, 
exit  airspeed,  and  beacon  code. 

ATCSs  indicated  that  airports,  sector  boundaries.  Instrument  Landing  System  (ILS)  approaches, 
restricted  area  boundaries,  and  ILS  outer-marker  information  were  most  important.  Less 
important  were  conflict  alert,  holding  pattern,  and  system  clock  information. 


Table  3  presents  detailed  information  on  the  ATCS  ratings  of  important  radarscope  information. 
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Table  2.  Importance  of  Aircraft  Liforaiation  (N  =  12) 


Variable  Label 

Mean! 

SDs 

Current  Altitude 

9.33 

0.89 

Current  Location 

9.33 

0.98 

Assigned  Altitude 

9.17 

1.03 

Arrival  Apt.  (within  sector) 

8.67 

1.50 

Call  Sign 

8.33 

3.45 

Departure  Apt.  (within  sector) 

8.25 

2.30 

Near  Exit  Fix/Arrival  Apt. 

8.17 

2.12 

7.92 

1.88 

Density 

7.92 

1.31 

Exit  Altitude 

7.58 

1.88 

Waiting  for  Hand-ofPRelease 

7.42 

2.15 

7.33 

1.56 

Current  Airspeed 

7.17 

1.75 

Assigned  Airspeed 

7.00 

1.48 

Current  Heading 

6.92 

1.93 

Entry  Altitude 

6.58 

2.97 

Exit  Fix 

6.58 

1.88 

ATCS  Ownership 

6.36 

3.80 

6.17 

2.25 

Entry  Airspeed 

5.58 

2.31 

Entry  Fix 

4.92 

2.57 

Exit  Airspeed 

4.75 

2.45 

Beacon  Code 

4.58 

3.26 

Table  3.  Importance  of  Radarscope  Information  (N  —  12) 


Variable  Label 


Airports 


Sector  Boundaries 


ILS  Approaches 


Restricted  Area  Boundaries 


ILS  Outer  Marker 


Runways 


Fixes 


VORs 


Future  Act.  List 


Range  Rings 


Obstructions 


Filter  Settings 


Conflict  Alert 


Holding  Patterns 


System  Clock 


Mean 


8.83 


8.83 


8.75 


8.58 


8.50 


7.75 


7.50 


7.42 


5.50 


5.33 


5.33 


5.33 


5.33 


4.67 


4.08 


SDs 


JA7 

JAO 

_L4i 

JJl 

_L6S 

2AS 

_2J5 

235 

_^3 

2^7 

_2A6 

_231 

JJO 

_2J0 

2.75 


3.2.2  Post-Scenario  Questionnaire 


The  Post-Scenario  Questionnaire  contained  eight  general  questions  concerning  realism, 
representativeness,  ATWIT  interference,  oculometer  interference,  simulation  pilot 
responsiveness,  working  hard,  quality  of  control,  and  difficulty.  Table  K-1,  Appendix  K, 
presents  the  means  and  SDs  for  these  questions. 

The  analysis  investigated  if  a  difference  in  ATCS  response  occurred  when  task  load  changed 
from  low  to  high  or  when  the  scenario  changed  from  having  no  visual  noise  to  having  visual 
noise.  If  the  analysis  showed  that  the  experimental  conditions  did  affect  the  general  questions 
significantly,  the  subsequent  analyses  consisted  of  ANOVAs  on  individual  variables. 

The  effects  of  increasing  task  load  and  introducing  visual  noise  on  the  responses  to  the  general 
post-scenario  questions  interacted  significantly  [A  =  .41,  F(8, 15)  =  2.66,  p  <  .05]  (Appendix  L, 
Table  L-1).  Because  of  this  interaction,  researchers  analyzed  visual  noise  impact  under  both  low 
and  high  task  loads  and  also  task  load  with  or  without  visual  noise.  The  effect  of  increasing  task 
load  on  responses  to  general  post-scenario  questions  was  slightly  stronger  in  the  absence  of 
visual  noise  [A  =  .04,  F(8,  15)  =  44.30  versus  A  =  .08,  F(8,  15)  =  22.08,p  <  .05,  or  q  =  .98 
versus  .96,  respectively].  The  effect  of  introducing  visual  noise  was  only  significant  under  high 
task  load  conditions  [A  =  .41,  F(8, 15)  =  2.65,  p  <  .05]. 

Because  the  MANOVA  results  indicated  that  the  experimental  conditions  affected  the  general 
post-scenario  questions,  researchers  analyzed  each  of  the  questions  individually.  To  maintain  an 
overall  alpha  level  of  .05,  the  researchers  adjusted  the  alpha  level  to  .0064  for  the  analyses. 
Without  the  adjustment  of  the  alpha  level,  the  sequence  of  subsequent  univariate  analyses  may 
allow  the  overall  probability  of  error  to  creep  upward.  Figure  4  presents  the  means  and  SDs  for 
the  eight  general  post-scenario  questions. 
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Figure  4.  General  post-scenario  questions  as  a  fimction  of  task  load  and  visual  noise. 
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a.  Realism  and  Representativeness 

Visual  noise  made  the  scenarios  slightly  more  realistic  although  not  statistically 
significant.  The  scenarios  were  equally  representative  of  an  average  day  at  the  TRACON. 
Although  not  statistically  significant,  ATCSs  indicated  that  the  low  task  load  scenarios 
were  more  realistic  than  the  high  task  load  scenarios. 

b.  ATWIT  and  Oculometer  Interference 

The  ATCS  perceived  little  interference  fi'om  the  ATWTT  device.  The  equipment 
bothered  them  even  less  when  the  task  load  was  low.  The  oculometer  hardly  interfered, 
but  more  than  the  ATWTT  device.  The  ATCSs  did  not  perceive  that  increased  task  load 
caused  any  greater  oculometer  interference.  Visual  noise  in  the  scenario  reduced  the 
perceived  level  of  interference  caused  by  the  oculometer,  although  not  significantly. 

c.  Simulation  Pilot  Responsiveness 

The  perceived  quality  of  the  simulation  pilot  responses  was  very  hi^.  Increasing  task 
load  reduced  the  perceived  quality  of  these  responses,  but  not  significantly.  Introducing 
visual  noise  did  not  alter  the  perceived  quality  of  the  responses. 

d.  Working  Hard 

The  effect  of  increasing  task  load  on  the  perception  of  ATCSs  on  how  hard  they  worked 
during  the  simulation  depended  on  the  presence  of  visual  noise  [F(l,  22)  =  9.24,  p  <  .05] 
(Table  L-2).  Researchers  determined  simple  effects.  ATCSs  felt  they  worked  harder 
during  high  task  load  scenarios  [F(l,  22)  =  296.66, p  <  .05].  The  increase  in  perceived 
workload  due  to  an  increase  in  task  load  was  smaller  when  visual  noise  was  present  than 
when  it  was  absent. 

e.  Quality  of  Control 

Participants  perceived  that  their  control  quality  was  lower  under  high  task  load  conditions 
[F(l,  22)  =  14.44,p  <  .05]  (Table  L-3).  Under  high  task  load  conditions,  visual  noise  led 
to  an  increase  in  perceived  quality  of  control,  although  not  statistically  significant.  Under 
low  task  load  conditions,  visual  noise  did  not  affect  the  perceived  quality  of  performmce. 
The  introduction  of  visual  noise  showed  a  trend  toward  an  increase  in  perceived  quality 
of  control,  althou^  not  significantly. 

f  Difficulty 

The  effects  of  increasing  task  load  and  introducing  visual  noise  on  perceived  simulation 
difficulty  interacted  [F(l,  22)  =  1 1.21,  p  <  .05]  (Table  L-2).  Visual  noise  itself  did  not 
affect  the  perceived  difficulty,  but  it  altered  the  effect  of  increasing  task  load.  Introducing 
visual  noise  increased  the  perceived  difficulty  under  low  task  load  conditions,  but  it 
reduced  the  perceived  difficulty  under  high  task  load  conditions. 
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g.  Situation  Awareness  Questions 

The  four  post-scenario  questions  involving  SA  estimates  included  overall  SA,  current 
aircraft  location  SA,  projected  aircraft  (A/C)  location  SA,  and  potential  violation  SA. 

The  post-scenario  questions  that  addressed  the  ATCSs’  SA  showed  a  multivariate 
significance  for  the  effects  of  increasing  task  load  [A  =  .32,  F(4, 19)  =  10.3  l,p  <  .05]  and 
introducing  visual  noise  [A  =  .55,  F(4, 19)  =  3.86,p  <  .05]  (Table  L-4).  The  MANOVA 
on  SA  related  questions  involved  responses  for  four  questions.  To  maintain  an  overall 
alpha  level  of  .05,  the  adjusted  alpha  level  for  the  analyses  on  individual  questions  was 
.013. 

The  ATCSs  estimated  their  SA  higher  under  low  task  load  than  under  high  task  load 
conditions  [Overall  SA,  F(l,  22)  =  25.19,  Current  A/C  Location  SA,  F(l,  22)  =  42.98, 
Projected  A/C  Location  SA,  F(l,  22)  =  32.85,  Potential  Violations  SA,  F(l,  22)  =  13.03, 
all  p  <  .05]  (Table  L-5).  Figure  5  summarizes  the  means  and  SDs. 
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Figure  5.  Means  and  SDs  for  SA  post-scenario  questions  as  a  function  of  task  load. 

Visual  noise  affected  only  the  SA  question  concerning  potential  violations  [F(l,  22)  =  14.63, 
p  <  .05]  (Table  L-6).  ATCSs  perceived  that  they  had  a  better  SA  for  potential  violations  (Figure 
6)  in  the  presence  of  visual  noise. 
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Figure  6.  Means  and  SDs  for  SA  for  potential  violations  as  a  function  of  visual  noise. 
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3.2.2.1  Post-Scenario  TLX 


The  items  of  the  NASA  TLX  were  mental,  physical,  and  temporal  demand;  performance;  effort; 
and  frustration.  The  MANOVA  on  these  items  displayed  a  significant  effect  of  increasing  task 
load  [A  =  .06,  F(6, 17)  =  45.17,p  <  .05].  To  ensure  an  overall  alpha  level  of  .05,  the  adjusted 
alpha  was  .0085  for  the  ANOVAs  on  all  six  items. 

The  mental,  physical,  and  temporal  demand;  level  of  effort;  and  fiiistration  were  higher  under 
high  task  load  conditions  than  low  task  load  conditions  [F(l,  22)  =  222.27, 41.91, 99.95, 23.84, 
80.05  respectively,  all  atp  <  .05].  The  performance  level  was  lower  under  high  task  load  than 
under  low  task  load  conditions  [F(l,  22)  =  8.72,p  <  .05].  Table  L-8  presents  detailed  ANOVA 
results  for  the  effect  of  task  load  Figure  7  presents  the  means  and  SDs  of  the  individual  TLX 
items. 
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Figure  7.  Means  and  SDs  for  post-scenario  TLX  items  as  a  function  of  task  load. 

3.2.3  Exit  Questionnaire 

After  the  eight  experimental  scenarios,  the  participants  completed  an  Exit  Questionnaire 
(Appendix  C).  The  Exit  Questionnaire  collected  their  opinions  on  topics  covered  in  the  Post- 
Scenario  Questionnaires.  The  ATCS  rated  each  item  on  a  scale  from  1  to  10.  The  overall 
realism  of  tiie  scenarios  was  moderately  good.  The  participants  perceived  the  scen^os  as  a 
moderately  realistic  representation  of  an  average  day  at  their  TRACON.  The  participants  felt  that 
the  ATWrr  device  hardly  interfered  with  controlling  traffic.  The  oculometer  interfered  more 
than  the  ATWIT  device,  but  the  level  of  interference  was  low.  The  simulation  pilots  performed 
extremely  well.  The  hands-on  training  was  adequate  (Table  4). 


Table  4.  Exit  Questionnaire  (N  =  12) 


Variable  Label 

Mean 

SDs 

Realism 

6.42 

1.44 

Representative 

5.67 

2.15 

ATWIT  interference 

1.58 

0.90 

Oculometer  interference 

3.17 

2.55 

Simulation  pilot  performance 

9.33 

Training  adequacy 

8.91 

1.14 
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3.3  Over-the-Shoulder  Evaluation 


3.3.1  Ratines 

The  OTS  rating  form  contained  three  sets  of  questions.  The  first  concerned  ATCS  performance. 
The  second  set  consisted  of  selected  items  from  the  Post-Scenario  Questionnaire.  The  third  set 
of  questions  included  the  six  items  of  the  NASA  TLX.  Researchers  analyzed  each  of  these 
groups  of  questions  separately. 

The  general  OTS  evaluation  consisted  of  questions  related  to  Maintaining  Safe  and  Efficient 
Traffic  Flow,  Maintaining  Attention  and  SA,  Prioritizing,  Providing  Control  Information, 
Technical  Knowledge,  and  Communication. 

Traffic  load  manipulation  affected  all  questions  related  to  Maintaining  Safe  and  Efficient  Traffic 
Flow.  Under  high  task  load  conditions,  the  OTS  rater  evaluated  maintaining  separation  and 
resolving  potential  conflicts  lower  and  ATCSs  sequenced  arrival  and  departure  aircraft  more 
efficiently  (Figure  8). 
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□  Low  Task  Load  ■  High  Task  Load 

Figure  8.  Means  and  SDs  for  traffic  flow  related  questions  as  a  function  of  task  load. 

Task  load  manipulation  affected  all  questions  related  to  Maintaining  Attention  and  SA  (Figure 
9).  With  increasing  task  load,  the  participants  maintained  awareness  of  aircraft  positions  less  but 
ensured  positive  control.  Also,  detection  of  pilot  deviations  from  control  instructions  was  less 
likely,  and  ATCSs  corrected  their  own  errors  in  a  less  timely  manner. 
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Figure  9.  Means  and  SDs  of  variables  related  to  maintaining  attention  and  SA  as  a  function  of 
task  load. 
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Task  load  manipulation  affected  all  questions  related  to  Prioritizing.  The  OTS  rater  indicated 
that  all  prioritizing-related  variables  showed  a  lower  performance  imder  high  task  load  (Figure 
10).  However,  mean  ratings  indicated  that  overall  observers  believed  performance  was  on  the 
top  third  of  the  scale. 


Tdcing  acdons  in  an  Prq)lanning  control  actions  Handling  control  task  for  Marking  fli^t  strips  while 

appropriate  order  of  several  MC  performing  other  tasks 

inqiortanoe 


□  Low  Task  Load  ■  High  Task  Load 


Figure  10.  Means  and  SDs  for  variables  related  to  prioritizing. 

The  visual  noise  manipulation  affected  prqjlanning  control  actions.  Participants  showed  better 
preplanning  when  visual  noise  was  present  than  when  visual  noise  was  absent  (Figure  1 1). 
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Figure  1 1.  Means  and  SDs  for  preplanning  control  actions  as  a  function  of  visual  noise. 

The  section  in  the  OTS  rater’s  form  on  Providing  Control  Information  provided  essential  ATC 
information.  An  increase  of  task  load  lowered  the  OTS  rater  perception  of  the  quality  of 
providing  essential  ATC  information  (Figure  12). 
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Figure  12.  Means  and  SDs  for  providing  essential  ATC  information  as  a  function  of  task  load. 

The  observer  perceived  a  decrease  in  providing  additional  ATC  information  as  task  load 
increased.  In  the  absence  of  visual  noise,  increasing  task  load  reduced  the  amount  of  additional 
ATC  information  provided  (Figure  13). 
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□  Low  Task  Load 
■  High  Task  Load 

No  Visual  Noise  Visual  Noise 

Figure  13.  Means  and  SDs  of  providing  additional  control  infoiniation  as  a  function  of  task  load 
and  visual  noise. 

The  questions  on  Technical  Knowledge  consisted  of  showing  knowledge  of  LOAs  and  SOPs  and 
showing  knowledge  of  aircraft  capabilities  and  limitations.  Neither  task  load  or  visual  noise 
affected  the  responses  to  these  questions. 

The  issues  related  to  the  quality  of  ATCS  Communications  were  using  proper  phraseology, 
communicating  clearly  and  efficiently,  and  listening  for  pilot  readbacks  and  requests.  Clarity, 
efficiency,  and  the  quality  of  listening  for  pilot  readbacks  decreased  with  increasing  task  load 
(Figure  14),  although  the  OTS  rater  did  not  notice  a  difference  in  the  use  of  proper  phraseology. 


Communicating  clearly  and  Listening  for  pilot  readbacks 
efdciently  and  requests 


Figure  14.  Means  and  SDs  for  variables  related  to  communication  as  a  fimction  of  task  load. 


Figure  15  presents  the  means  and  SDs  of  the  six  NASA  TLX  items,  which  are  the  observer’s 
estimates  of  participant  workload  dimensions.  An  increase  in  task  load  increased  the  perceived 
level  of  Mental  Demand,  Frustration,  Physical  Demand,  Temporal  Demand,  and  Effort.  The 
presence  of  visual  noise  reduced  the  task  load  effects  for  Mental  Demand  and  on  Frustration  and 
lowered  the  level  of  Performance  under  high  task  load. 
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Figure  15.  Means  and  SDs  of  OTS  NASA  TLX  items  by  task  load  and  visual  noise. 

^  7  rnmments  Related  to  Class  C  Airspace  Violations 

The  OTS  rater  comments  provided  valuable  information  about  how  ATCSs  dealt  with  the 
incursions.  According  to  FAA  Order  71 10.65J  (FAA,  1996),  ATCSs  must  attempt  to  establish 
two-way  radio  communications  with  any  aircraft  entering  Class  C  airspace.  This  study  revealed 
that  only  a  few  ATCSs  correctly  followed  this  order.  The  descriptions  below  are  summaries  of 
the  comments  on  the  four  questions  related  to  controller  SA  made  by  the  OTS  rater. 

Scenario  1,  a  low  task  load  simulation  with  visual  noise  present,  contained  one  VFR  incmrsion 
and  one  IFR  incursion.  The  VFR  incursion  flew  through  Class  C  airspace  at  2,500  feet.  The  IFR 
incursion  skimmed  the  top  of  Class  C  airspace  at  7,000  feet.  Several  of  the  ATCSs  did  not 
acknowledge  the  presence  of  one  or  both  of  the  intruders.  The  ATCSs  that  did  recognize  the 
incursion  of  their  airspace  displayed  a  wide  variety  of  actions  after  the  detection  of  an  incursion. 
The  ATCS  often  recognized  the  VFR  intruders,  issued  the  intruder  as  ftaffic  to  other  aircraft,  but 
did  not  attempt  to  establish  two-way  communications.  Other  ATCSs  called  local  control  or  the 
tower  to  inform  them  about  the  presence  of  a  VFR  intruder  in  Class  C  airspace.  Actions  taken 
after  detecting  the  IFR  intruder  ranged  from  calling  the  ARTCC  for  information  about  the 
aircraft,  to  attempting  to  establish  two-way  radio  communications. 

Scenario  2,  a  high  task  load  simulation  with  visual  noise  present,  contained  one  VFR  and  one 
IFR  Class  C  airspace  incursion.  The  VFR  incursion  aircraft  took  off  from  an  airport  just  outside 
of  Class  C  airspace  and  flew  into  Charlie  airspace  at  2,500  feet.  The  IFR  incursion  aircraft 
descended  from  high  altitude  into  Class  C  airspace  without  announcing  itself.  Before  it  became  a 
Class  C  violator,  the  aircraft  contained  neither  a  limited  nor  a  full  data  block.  Several  of  the 
ATCSs  failed  to  detect  the  incursions  into  Class  C  airspace.  The  observer  indicated  that  ‘‘most  of 
the  time,  the  intruder’s  limited  data  block  was  near  the  full  data  block  of  another  aircraft.”  Some 
ATCSs  noticed  the  incursions  and  took  jq)propriate  action.  They  called  adjacent  sectors,  tried  to 
establish  two-way  radio  communications,  and  issue  the  intruder  as  traffic  when  appropriate. 
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Scenario  3,  a  low  task  load  scenario  without  visual  noise,  contained  two  VFR  intruders.  One  of 
the  intruders  entered  Class  C  airspace  at  3,000  feet.  The  other  intruder  did  not  actually  enter 
Class  C  airspace  but  was  traffic  for  other  aircraft.  Most  of  the  ATCSs  recognized  the  VFR 
incursion  into  Class  C  airspace,  and  several  of  them  coordinated  with  the  tower  or  issued  the 
intruder  as  traffic  to  other  aircraft. 

Scenario  4,  a  high  task  load  scenario  without  visual  noise,  contained  two  VFR  incursions  of 
Class  C  airspace.  This  simulation  contained  two  VFR  intruders.  The  first  intruder  entered  Class 
C  airspace  at  3,500  feet  from  a  southwest  direction.  The  other  intruder  entered  Class  C  airspace 
at  2,500  feet  from  a  northeast  direction.  Several  of  the  ATCSs  did  not  acknowledge  one  of  the 
intruders  as  it  flew  throu^  Class  C  airspace  even  when  it  passed  near  other  aircraft  as  traffic. 
Some  of  these  ATCSs  recognized  an  intruder  only  after  it  passed  through  Class  C  airspace. 

Other  ATCSs  saw  both  intruders  and  issued  them  several  times  as  traffic  to  other  aircraft. 

Scenario  5,  a  low  task  load  simulation  with  visual  noise  present,  contained  two  IFR  Class  C 
incursions.  The  first  incursion  descended  from  9,000  feet  to  7,000  feet  (the  ceiling  of  Class  C 
airspace  for  this  TRACON)  and  came  in  from  a  north/northeast  direction.  The  second  incursion 
descended  from  8,500  feet  to  7,000  feet  from  a  southwest  direction.  Both  IFR  intruders  were 
part  of  the  high  altitude  overflights  that  simulated  the  visual  noise.  Before  becoming  an  intruder, 
the  aircraft  contained  neither  limited  nor  full  aircraft.  Some  of  the  ATCSs  did  not  detect  one  or 
both  of  the  intruders,  although  the  traffic  load  was  light.  Other  ATCSs  noticed  an  intruder  only 
after  it  had  passed  through  Class  C  airspace.  The  response  of  ATCSs  that  noticed  the  intruders 
varied  from  calling  adjacent  sectors  to  inquire  about  aircraft,  to  establishing  two  way 
communications,  and  to  issuing  traffic  when  appropriate. 

Scenario  6,  a  high  task  load  scenario  with  visual  noise  present,  contained  two  IFR  Class  C 
airspace  incursions.  This  simulation  contained  two  IFR  intruders  that  dropped  from  a  higher 
altitude  down  to  2,000  feet  into  Class  C  airspace  from  a  South/South-West  direction.  The  OTS 
rater  indicated  that  many  of  the  ATCSs  did  not  notice  one  or  both  of  the  IFR  incursions  into 
Class  C  airspace.  In  some  cases,  an  ATCS  detected  an  intruder  after  it  had  passed  through  Class 
C  airspace.  (The  intruder  was  finally  identified  about  10  miles  before  exiting  the  airspace).  In 
this  high  task  load  scenario,  several  controllers  had  operational  errors  that  involved  an  IFR 
intruder.  (The  second  intruder  merged  with  another  aircraft  at  3,500  feet  without  a  traffic 
advisory  being  issued).  Some  of  the  ATCSs  detecting  one  or  both  of  the  IFR  incursions 
contacted  the  tower,  but  other  ATCSs  did  not  take  further  action. 

To  assess  how  many  ATCSs  missed  intrusions,  researchers  reviewed  the  OTS  rater  comments 
and  tallied  the  number  of  intrusions  the  ATCS  issued  as  traffic,  inquired  with  other  facilities 
about,  tried  to  contact,  or  otherwise  acknowledged  the  intruder.  Figure  16  presents  the  results. 

Of  the  eight  scenarios,  six  included  incursions  into  Class  C  airspace.  Four  of  these  scenarios 
contained  high  altitude  overfli^ts  as  visual  noise.  There  were  three  scenarios  of  each  task  load 
level.  Although  the  number  of  observations  was  not  equally  distributed  across  conditions, 
researchers  calculated  the  proportion  of  controllers  that  either  missed  both  incursions,  picked  up 
one  of  the  incursions,  or  picked  up  both  incursions  (Figure  16).  In  each  of  the  conditions,  at  least 
1  of  the  12  participating  ATCSs  missed  one  of  the  intruders.  In  the  extreme  case  of  high  task 
load  and  presence  of  visual  noise,  one  fifth  of  the  ATCSs  detected  both  intruders. 
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Figure  16.  Percent  of  ATCSs  that  indicated  detection  of  the  Class  C  airspace  violations. 

3.4  Visual  Scanning 

The  summary  variables  for  5-minute  intervals  formed  the  basis  for  the  visual  scanning  data  set. 
The  5-minute  intervals  enabled  rejection  of  a  single  interval  without  loosing  the  complete 
simulation.  Researchers  replaced  the  variable  values  for  that  rejected  interval  with  the  average 
values  across  all  conditions  for  that  interval.  Of  864  intervals  (12  participants  x  8  scenarios  x  9 
intervals),  the  researchers  rejected  15  intervals  due  to  a  low  number  of  saccades  (less  than  200 
saccades  in  a  5-mmute  interval)  and  10  intervals  due  to  a  high  number  of  saccades  (more  Hhm 
800  saccades  in  a  5-minute  interval).  For  all  rejected  intervals,  researchers  substituted  the  visual 
scanning  variables  by  overall  5-minute  interval  means.  Therefore,  the  number  of  summary  data 
points  presented  in  the  Results  Section  is  864  [based  on  12  (participants)  x  8  (scenarios)  x  9 
(intervals)  =  864].  The  5-minute  interval  data  formed  the  basis  for  the  summary  data  per 
scenario. 

The  visual  scanning  variables  represented  three  levels  of  detail.  The  &st  level  included  general 
characteristics  of  fixations,  saccades,  blinks  and  pupil  diameter.  The  second  level  included 
characteristics  of  fixations  by  scene  plane:  the  radarscope,  the  ATWTT  panel,  the  flight  progress 
strip  bay,  and  the  keyboard/mouse  area.  The  third  level  included  characteristics  of  fixations  on 
radarscope  objects:  aircraft,  low  altitude  and  conflict  alert  areas,  system  area,  tab  list,  static 
objects  (airport,  runways,  fixes,  VORs),  and  preview  area.  The  following  sections  discuss  each 

of  the  levels. 
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3 .4. 1  General  Eve  Movement  Characteristics 


Variables  reflecting  general  eye  movement  characteristics  included  fixations,  saccades,  blinks, 
and  pupil  diameter.  The  variables  used  to  analyze  differences  in  general  eye  movement 
characteristics  between  conditions  were 

a.  number  of  fixations, 

b.  mean  fixation  duration, 

c.  mean  fixation  area, 

d.  visual  efficiency, 

e.  mean  saccade  duration, 

f.  mean  saccade  distance, 

g.  eye  motion  workload, 

h.  mean  pupil  diameter, 

i.  motion  workload, 

j.  number  of  blinks, 

k.  mean  blink  diiration,  and 

l.  mean  blink  distance. 

Appendix  F  presents  definitions  for  several  of  the  general  eye  movement  variables.  Appendix  L, 
Table  L-12  presents  the  results  of  the  MANOVA.  The  only  effect  on  general  eye  movement 
characteristics  was  due  to  the  task  load  manipulation  [A  =  .35,  F(5, 18)  =  6.68,p  <  .05].  The 
reader  should  bear  in  mind  that  the  DVs  used  in  the  multivariate  analyses  are  somewhat 
correlated.  The  correlations  between  Ihe  DVs  used  in  the  multivariate  analysis  do  not  reach  a 
level  where  one  of  the  variables  is  redxmdant.  Table  L-13  shows  the  details  of  the  ANOVA 
results  for  the  effect  of  task  load  on  general  eye  movement  characteristics. 

To  maintain  an  overall  alpha  of  .05  with  1 1  DVs,  the  adjusted  alpha  used  in  the  univariate 
ANOVAs  was  .0047.  Increasing  task  load  or  introducing  vistial  noise  did  not  affect  the  number 
of  fixations.  Only  mean  fixation  area  showed  a  significant  increase  between  the  low  and  the  high 
task  load  conditions  [F(l,  22)  =  19.54,p  <  .05]  (Figure  17).  Although  introducing  visual  noise 
affected  how  much  the  fixation  area  increased  with  task  load,  this  interaction  did  not  reach 
statistical  significance. 
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Figure  17.  Means  and  SDs  of  fixation  area  as  a  fimction  of  task  load  and  visual  noise. 

Although  saccade  distance  decreased  as  a  fimction  of  task  load,  it  did  not  reach  statistical 
significance  (Figure  18), 
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Figure  18.  Means  and  SDs  of  the  saccade  distance  as  a  fimction  of  task  load  and  visual  noise. 
3.4.2  Scene  Plane  Fixations 

The  scene  plane  fixation  variables  included  the  number  and  duration  of  fixations  on  the 
radarscope,  fli^t  strip  bay,  ATWIT  device,  and  keyboard  area.  The  MANOVA  results  showed 
an  interaction  between  load  and  visual  noise  [A  =  .25,  F(4, 19)  =  14.20,p  <  .05]  on  scene  plane 
fixation  characteristics  (Table  L-14). 

To  maintain  an  overall  alpha  level  of  .05  with  eight  variables,  researchers  used  the  adjusted  alpha 
level  of  .00639.  Table  L-15  presents  the  ANOVA  results  for  the  interaction  between  the  effects 
of  task  load  and  visual  noise. 

The  introduction  of  visual  noise  interacted  significantly  with  the  effect  of  increasing  task  load  on 
the  number  of  fixations  on  the  radarscope  [^(1, 22)  =  15.62,p  <  .05].  The  number  of  fixations 
on  the  radarscope  within  a  scenario  was  higher  when  task  load  was  low.  The  number  of  fixations 
on  the  radarscope  was  larger  when  visual  noise  was  present  under  low  and  smaller  under  high 
task  load  conditions  (Figure  19). 
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Figure  19.  Mean  and  SD  of  the  total  number  of  fixations  on  the  radarscope  as  a  fimction  of 
visual  noise  and  load  over  a  45-minute  scenario. 

Increasing  task  load  and  introducing  visual  noise  interacted  for  duration  of  fixations  on  the 
radarscope  [F(l,  22)  =  17.49, p  <  .05].  The  mean  fixation  duration  on  the  radarscope  in  the 
absence  of  visual  noise  was  higher  for  low  task  loads  than  for  high  task  loads.  The  presence  of 
visual  noise  reversed  this  effect,  and  the  mean  fixation  duration  increased  under  high  task  load 
conditions  (Figure  20). 


□No  Visual  Noise 
■  Visual  Noise 


Low  Task  Load  High  Task  Load 

Figure  20.  Mean  and  SD  of  fixation  duration  on  the  radarscope  as  a  fimction  of  visual  noise  and 
task  load. 

The  load  and  visual  noise  interaction  effect  for  the  number  of  fixations  on  the  fli^t  strip  bay 
[F(l,  22)  =  14.72,p  <  .05]  was  significant.  The  number  of  fixations  on  the  flight  strip  bay  stayed 
the  same  under  low  and  high  task  load  conditions  when  visual  noise  was  absent.  When  visual 
noise  was  present,  the  number  of  fixations  on  the  flight  strip  bay  increased  under  high  task  load 
conditions.  When  the  task  load  was  low,  the  introduction  of  visual  noise  changed  the  number  of 
fixations  on  the  flight  strip  bay  only  marginally.  Under  high  task  load,  the  introduction  of  visual 
noise  introduced  a  substantial  increase  in  the  number  of  fixations  on  the  flight  strip  bay  (Figure 
21). 
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Figure  21.  Mean  and  SD  of  number  of  fixations  on  the  flight  strip  bay  as  a  fimction  of  visual 
noise  and  task  load. 

Task  load  and  visual  noise  manipulation  did  not  interact  for  the  duration  of  fixations  on  the  flight 
strip  bay.  The  fixations  were  significantly  shorter  in  duration  for  high  task  load  conditions  than 
for  low  task  load  conditions  [F(l,  22)  =  36.95,  p  <  .05]  (Figure  22). 
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Figure  22.  Mean  and  SD  of  the  mean  fixation  duration  on  the  flight  strip  bay  as  a  function  of 
task  load. 

Increasing  task  load  significantly  increased  the  number  of  fixations  on  the  keyboard  area 
[F(l ,  22)  =131.55,p<  .05]  (Figure  23).  The  number  of  fixations  on  the  keyboard  area  increased 
'  by  approximately  41%.  Increasing  task  load  or  introducing  visual  noise  did  not  affect  the 
number  or  the  duration  of  fixations  on  the  ATWIT  device  or  the  fixation  duration  on  the 
keyboard  area. 


Low  Task  Load  High  T ask  Load 


Figure  23.  Means  and  SDs  of  the  number  of  fixations  on  the  keyboard  area  as  a  function  of  task 
load  and  visual  noise. 

3.4.3  Radarscope  Fixations 

The  changes  in  the  fixation  characteristics  on  objects  on  the  radarscope  due  to  task  load  and 
visual  noise  were  not  indqiendent  [A  = .  15,  F(l,  22)  =  19.20,  p  <  .05]  (Table  L-16).  Because  of 
the  interaction  between  visual  noise  and  task  load  increase,  researchers  calculated  multivariate 
simple  effects.  The  alpha  level  after  adjusting  for  the  10  DVs  to  maintain  an  overall  alpha  of  .05 
was  .0051. 

The  effects  of  increasing  task  load  and  introducing  visual  noise  on  the  number  of  fixations  on  the 
system  area  influenced  one  another  [/^(l,  22)  =  10.54, p  <  .05]  (Table  L-17).  There  were  fewer 
fixations  on  the  system  area  under  high  task  load.  Introducing  visual  noise  reduced  the  number 
of  fixations  on  the  system  area.  This  reduction  was  less  pronounced  imder  high  task  load 
conditions  (Figure  24). 
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Figure  24.  The  number  of  fixations  on  the  systems  area  as  a  function  of  task  load  and  visual 
noise. 

Increasing  task  load  resulted  in  a  significant  [F(l,  22)  =  44.09, <  .05]  decrease  in  the  fixation 
duration  on  the  system  area  (Figure  25  and  Appendix  L,  Table  L-1 8).  Introducing  visual  noise 
did  not  significantly  alter  the  duration  of  fixations  on  the  system  area. 
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Figure  25.  Mean  fixation  duration  on  the  systems  area  as  a  function  of  task  load. 

The  mean  number  of  fixations  on  static  objects  showed  an  interaction  effect  of  the  manipulation 
of  task  load  and  visual  noise  [F(l,  22)  =  58.26,  p  <  .05].  Under  high  task  load  conditions, 
introducing  visual  noise  did  not  significantly  change  the  number  of  fixations  on  the  system  area. 
Under  low  task  load  conditions  introducing  visual  noise  significantly  reduced  the  number  of 
fixations  on  static  objects  (Figure  26).  ATCSs  spent  more  time  scanning  moving  objects  when 
visual  noise  was  present,  but  the  number  of  aircraft  under  control  was  low.  The  impact  of  these 
overflight  aircraft  targets  on  scanning  is  less  when  ATCSs  are  already  busy  with  more  demanding 
traffic  for  which  they  are  responsible. 


Figure  26.  Mean  number  of  fixations  on  static  objects  as  a  function  of  task  load  and  visual  noise. 

The  effects  of  the  introduction  of  visual  noise  and  the  increase  of  task  load  on  the  duration  of 
fixations  on  static  objects  interacted  [F(l,  22)  =  12.91, p  <  .05].  Under  low  task  load  conditions, 
the  fixation  duration  was  longer  when  visual  noise  was  absent.  Under  high  task  load  conditions, 
the  fixation  duration  increased  with  the  introduction  of  visual  noise  (Figure  27).  ATCSs  fixated 
on  fewer  objects  for  longer  periods.  The  visual  noise  introduced  a  need  to  be  more  selective  and 

concentrate  more. 
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Figure  27.  Mean  fixation  duration  on  static  objects  as  a  function  of  task  load  and  visual  noise. 

The  mean  number  of  fixations  on  the  tab  list  showed  an  interaction  between  the  task  load  and  the 
visual  noise  manipulation  [i^(l,  22)  =  20.85,/?  <  .05].  In  the  absence  of  visual  noise,  increasing 
task  load  led  to  a  reduction  of  fixations  on  the  tab  list.  The  presence  of  visual  noise  reversed  this 
effect  (Figure  28). 
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Figure  28.  Mean  number  of  fixations  on  tab  list  as  a  function  of  task  load  and  visual  noise. 

The  mean  duration  of  fixations  on  the  tab  list  did  not  change  significantly  between  conditions. 
Shorter  fixation  duration  imder  low  task  load  conditions  in  the  presence  of  visual  noise  showed  a 
trend,  but  it  was  not  statistically  significant  (Figure  29). 
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Figure  29.  Mean  fixation  duration  on  tab  list  as  a  function  of  task  load  and  visual  noise. 

The  mean  number  of  fixations  on  the  preview  area  did  not  show  a  significant  interaction  between 
increasing  task  load  and  introducing  visual  noise.  An  increase  in  task  load  led  to  a  significant 
[F(l,  22)  =  13.70,/?  <  .05]  reduction  of  the  number  of  fixations  on  the  preview  area  (Figure  30). 
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Figure  30.  Mean  number  of  fixations  on  preview  as  a  fimction  of  task  load. 

Introducing  visual  noise  led  to  a  significant  [F(l,  22)  =  26.40,  p  <  .05]  reduction  in  the  number  of 
fixations  on  the  preview  area  (Figure  31). 
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Figure  31.  Mean  number  offixations  on  preview  as  a  fimction  of  visual  noise. 

Researchers  could  not  study  the  effects  of  task  load  and  visual  noise  manipulation  on  the  number 
of  fixations  on  aircraft  independently  because  they  interacted  signific^tly 
[F(l,  22)  =  46.85,/?  <  .05].  Under  low  task  load  conditions,  introducing  visual  noise  did  not 
significantly  change  the  number  of  fixations  on  aircraft.  Under  high  task  load  conditions, 
introducing  visual  noise  reduced  the  number  of  fixations  on  aircraft  (Figure  32). 
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Figure  32.  Mean  number  of  fixations  on  aircraft  as  a  fimction  of  task  load  and  visual  noise. 
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An  interaction  between  task  load  and  visual  noise  manipulation  existed  for  the  fixation  duration 
[F(l,  22)  =  28.22,  j?  <  .05].  Introducing  visual  noise  under  low  task  load  conditions  led  to  a 
reduction  in  the  mean  duration  of  fixations.  Under  high  task  load  conditions,  introducing  visual 
noise  resulted  in  an  increase  in  the  mean  fixation  duration  (see  Figure  33). 
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Figure  33.  Mean  fixation  duration  on  aircraft  as  a  function  of  task  load  and  visual  noise. 

3.4.4  hitrusions 
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The  scenarios  for  each  participant  included  six  VFR  and  six  IFR  intrusions.  The  researchers 
isolated  the  5-minute  intervals  that  included  an  intrusion  for  the  analysis  of  eye  movements.  The 
study  contained  2  (load)  x  2  (visual  noise)  x  2  (replication)  scenarios.  The  analyses  compared 
the  intervals  that  included  intrusions  with  intervals  of  the  scenario  without  intrusions  that 
replicated  the  conditions.  For  five  of  the  VFR  and  IFR  intrusions,  such  an  interval  existed.  For 
the  other  interval,  the  VFR  intrusion  coincided  with  an  interval  that  contained  an  IFR  intrusion  in 
the  replication  scenario. 

The  research  team  conducted  repeated  measures  ANOVAs  on  the  DVs.  At  a  .05  level  of 
significance,  there  was  only  an  interaction  between  the  effect  of  the  presence  of  intrusions  and 
the  task  load  and  visual  noise  conditions  for  saccade  duration  (Table  L-19).  To  maintain  an 
alpha  level  of  .05  with  12  DVs,  researchers  reduced  the  adjusted  alpha  level  to  .0043.  At  this 
level,  the  effects  of  conditions  and  presence  of  intrusions  on  eye  movements  do  not  interact. 
There  was  no  effect  of  intrusions  on  any  of  the  general  eye  movement  characteristics  (Table  L- 
20).  The  data  pooling  procedures  may  have  washed  out  any  existing  effects. 

3.4.5  Radarscope  Objects 

The  researchers  tested  the  significance  of  the  difference  between  fixation  duration  on  several 
radarscope  objects  using  a  measure  called  “object  type.”  The  analysis  showed  the  presence  of 
higher  order  interactions  (up  to  the  three  way  interaction  between  objects,  load,  and  visual  noise 
[A  =  .56,  F(l,  22)  =  3.57, p  <  .05]  (Table  L-2 1).  The  mean  fixation  duration  on  radarscope 
objects  differed  significantly  for  each  of  the  task  load  and  visual  noise  conditions.  The  aircraft 
fixations  have  the  highest  durations  with  a  mean  of  655  ms  (Figure  34).  For  a  discussion  of  the 
effects  of  task  load  manipulation  and  visual  noise  on  the  individual  radarscope  objects  refer  to 
Section  3.4.3. 
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Figure  34.  Mean  and  SD  of  radar  object  fixation  duration  (ms)  as  a  fimction  of  task  load  and 
visual  noise. 


The  number  of  fixations  varied  significantly  between  objects.  The  effects  of  both  increasing  task 
load  and  introducing  visual  noise  significantly  interacted  with  the  effect  of  object  on  the  number 
of  fixations.  The  emphasis  on  aircraft  representations  becomes  even  clearer  when  presenting  file 
time  spent  on  radarscope  objects  as  a  percentage  of  file  total  time  (45  minutes).  Compared  to  the 
time  spent  on  aircraft  representations,  the  ATCS  allocates  a  negligible  amoimt  of  time  for  the 
other  objects.  ATCSs  spent  about  55%  of  the  total  simulation  time  on  fixating  aircraft 
representations.  Figure  35  displays  the  percentage  of  time  spent  on  radarscope  objects.  The 
figure  does  not  display  the  data  point  for  aircraft  to  allow  the  reader  to  compare  the  percentages 
between  objects  other  than  aircraft. 
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Figure  35.  Percent  of  total  simulation  time  fixated  on  selected  radarscope  objects. 

3.5  Performance  Measures 

The  performance  measures  used  in  the  analyses  consisted  of  conflicts,  errors,  commumcations, 
and  task  load-related  variables.  The  following  sections  will  discuss  each  of  the  categories  of 
variables. 

3.5.1  Conflicts 

The  DR&A  module  identifies  variables  in  this  section  as  conflict  related  based  on  ffR.  hi  the 
simulations,  both  IFR  and  VFR  aircraft  were  present.  The  conflict-related  variables  do  not 
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necessarily  reflect  the  occurrence  of  operational  errors.  The  conflict  data  calculated  on  EFR 
caused  the  DR&A  module  to  report  VFR  aircraft  being  in  conflict  when  no  conflict  existed.  This 
report  contains  information  about  conflict-related  variables  with  the  caveat  that  they  reflect  a 
flatness  of  control,  not  necessarily  a  reflection  of  operational  errors.  The  following  sections 
contain  descriptive  analyses  of  the  conflict-related  variables. 

The  number  of  standard  terminal  conflicts  increased  with  an  increase  in  task  load.  The  presence 
of  visual  noise  strengthened  this  effect.  The  effect  of  visual  noise  reduced  the  number  of 
standard  terminal  conflicts  under  low  task  load,  but  high  task  load  reversed  this  effect  (Figure 
36). 
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Figure  36.  Means  and  standard  deviations  for  number  of  standard  conflicts  as  a  function  of  task 
load  and  noise. 


Neither  load  nor  noise  affected  the  mean  duration  of  standard  conflicts.  Under  high  task  load 
conditions,  noise  increased  the  number  of  between-sector  conflicts,  hi  the  absence  of  visual 
noise,  task  load  manipulation  increased  the  number  of  between-sector  conflicts.  The  presence  of 
visual  noise  reduced  this  effect  (Figure  37). 
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Figure  37.  Mean  number  of  between-sector  conflicts  as  a  function  of  task  load  and  visual  noise. 


Under  low  task  load  conditions,  the  presence  of  visual  noise  did  not  affect  the  duration  of 
between-sector  conflicts.  Under  hi^  task  load  conditions,  visual  noise  increased  the  duration  of 
between-sector  conflicts.  The  manipulation  of  task  load  affected  the  duration  of  between-sector 
conflict  when  visual  noise  was  absent  and  present.  The  presence  of  visual  noise  increased  the 
duration  of  between-sector  conflicts  (Figure  38). 
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Figure  38.  Mean  duration  of  between-sector  conflicts  as  a  function  of  task  load  and  visual  noise. 
3.5.2  Separation 

Separation-related  variables  reflect  the  tightness  of  control.  The  analysis  includes  closest  point- 
of-approach,  the  horizontal  and  vertical  separation,  and  the  aircraft-proximity-index. 

The  repeated  MANOVA  showed  an  interaction  between  the  effects  of  task  load  and  visual  noise 
manipulation  on  separation-related  variables  [A  =  .50,  F(4, 19)  =  A.12,p  <  .05].  The  effect  of 
visual  noise  was  not  present  under  low  task  load  conditions.  Under  hi^  task  load  conditions, 
visual  noise  significantly  affected  separation  [A  = .  1 1,  F(4, 19)  =  40.20,;;  <  .05].  In  the  absence 
of  visual  noise,  there  was  a  small  effect  of  task  load  manipulation  on  separation 

19)  =  3,35, p  <  .05,  ti  =  .64].  In  the  presence  of  visual  noise,  there  was  a  stronger 

effect  of  task  load  manipulation  [A  =  .51,  F(4, 19)  =  4.57,p  <  .05,  t]  =  .70]. 

To  maintain  an  overall  alpha  level  of  .05  with  four  DVs,  the  adjusted  alpha  for  the  univariate 
analyses  is  .0127.  The  manipulation  of  task  load  had  a  sigmficant  effect  on  the  closest  point-of- 
approach  [F(l,  22)  =  13.37, p  <  .05]  (Figure  39). 
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Figure  39.  Mean  closest-point-of-approach  (feet)  as  a  function  of  task  load  and  visual  noise. 

Task  load  significantly  decreased  the  horizontal  separation  [F(l,  22)  =  13.03,p  <  .05].  Visual 
noise  did  not  affect  the  horizontal  separation  (see  Figure  40). 
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Figure  40.  Mean  horizontal  separation  as  a  function  of  task  load  and  visual  noise. 
3.5.3  Communications 
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Communications-related  variables  included  the  number  of  ATCS  messages  and  pilot  message 
keystrokes.  Task  load  manipulation  only  affected  communications  [F(2, 21)  =  217.33, p  <  .05]. 

With  only  two  DVs  used  in  the  MANOVA,  the  adjusted  alpha  level  to  be  used  in  subsequent 
ANOVAs  is  .025  to  maintain  an  overall  alpha  level  of  .05. 

The  number  of  ATCS  messages  showed  a  significant  increase  with  an  increase  of  task  load 
[F(l,  22)  =  54.10  and  F(l,  22)  =  103.72,  both  atp  <  .05]  (Figure  41).  The  presence  of  visual 
noise  did  not  significantly  affect  the  number  of  ATCS  messages. 
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Figure  41.  Mean  number  of  ATCS  messages  as  a  function  of  task  load  and  visual  noise. 

The  number  of  simulation  pilot  message  keystrokes  showed  a  significant  increase 

[F(l,  22)  =  103.72,p  <  .05]  with  an  increase  in  task  load  (Figure  42).  The  presence  of  visual 

noise  did  not  significantly  affect  the  number  of  simulation  pilot  message  keystrokes. 
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Figure  42.  Mean  ntunber  of  pilot  message  keystrokes  as  a  function  of  task  load  and  visual  noise. 
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3.5.4  Task  Load 


The  task  load  related  variables  showed  the  effect  of  the  task  load  manipulation.  These  variables 
did  not  provide  further  insight  in  the  effect  of  the  conditions  on  ATCS  performance  and  did  not 
undergo  further  analysis.  The  task  load  related  variables  did  not  go  further  statistical  analysis. 

4.  Discussion 

The  discussion  addresses  the  representativeness  of  the  simulations,  the  effect  of  increasing  task 
load  and  introducing  visual  noise  on  workload  measures,  the  effect  of  increasing  task  load  and 
introducing  visual  noise  on  SA  measures,  and  the  effect  of  a  task  load  and  visual  noise  on  eye 
movements.  Appendix  M  discusses  the  potential  for  alternative  analyses  with  the  format  of  the 
as  collected  during  the  current  experiment.  Appendix  N  contains  recommendations  for 
modifications  to  data  reduction  algorithms  and  future  research. 

4. 1  The  Representativeness  of  the  Scenarios 

A  high  level  of  fidelity  of  the  scenarios  allows  application  of  the  experimental  findings  to  an 
operational  setting.  Researchers  designed  representative  scenarios  of  an  active  TRACON.  The 
TRACON  radar  display  shows  aircraft  under  control  or  within  the  filter  limits  and  the  raw  radar 
returns  of  aircraft  outside  the  filter  lunits.  The  ATCSs  acknowledged  the  high  fidelity  of  the 
scenarios  by  positively  rating  the  realism  and  representativeness  of  the  scenarios.  The  Post- 
Scenario  Questionnaires  indicated  that,  on  average,  the  scenarios  were  moderately  realistic  and 
representative  of  a  normal  day  at  their  TRACON.  Scenarios  were  only  moderately  difficult, 
which  is  an  indication  that  the  low  and  high  task  load  scenarios  were  well  balanced.  The 
interference  of  the  oculometer  was  low  although  higher  than  the  interference  of  the  ATWIT 
device. 

4.2  The  Effect  of  Time-on-Task.  Task  Load,  and  Visual  Noise  on  W orkload  Measures 

The  effect  of  task  load  manipulation  was  stronger  without  visual  noise  than  when  visual  noise 
was  present.  ATCSs  rated  all  TLX  items  except  performance  higher  when  task  load  increased. 
The  rating  for  the  performance  item  decreased  with  increasing  task  load.  Although  OTS 
observations  showed  an  interaction  between  the  effects  of  increasing  task  load  and  introducing 
visual  noise,  they  corresponded  well  with  ATCSs’  own  ratings.  These  findings  are  common  in 
studies  using  self-reported  workload.  Perceived  performance  declines  at  higher  levels  of 
workload  given  professional  respondents  who  are  trying  to  accurately  gauge  their 
accompUshments. 

The  average  ATWIT  rating  as  a  function  of  time  showed  the  effect  of  the  sfiucture  in  the 
scenarios  used  in  this  study.  The  traffic  in  these  scenarios  increased  in  the  first  10  minutes  and 
tapered  down  at  the  end  of  the  45-minute  scenarios.  On  average,  the  ATWIT  ratings  reflected 
this  trend.  ATCSs  rated  the  workload  low  in  the  beginning  of  the  scenarios,  increasing  up  to  the 
third  5-minute  interval,  and  decreasing  somewhat  at  the  end  of  the  scenarios.  Only  task  load 
affected  the  mean  ATWIT  scores.  The  high  task  load  scenarios  resulted  in  a  higher  perceived 
workload.  Visual  noise  had  no  effect  on  the  mean  ATWIT  ratings.  The  effect  of  task  load 
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resulted  in  a  higher  maximum  ATWIT  rating,  and  the  presence  of  visual  noise  resulted  in  an 
increased  contrast  between  low  and  high  task  load  conditions. 

The  disadvantage  of  using  post-scenario  estimates  of  the  perceived  workload  during  a  scenario  is 
that  the  ATCS  has  to  rely  on  memory  for  the  workload  across  a  45-minute  period.  To  investigate 
if  an  ATCS  remembers  the  average  or  the  maximum  workload  perceived  during  a  scenario, 
researchers  computed  the  correlations  between  the  average  and  maximum  on-line  ATWIT  ratings 
with  the  post-scenario  TLX  items.  The  TLX  item  on  mental  demand  showed  the  highest 
correlation  with  the  average  ATWIT  rating,  explaining  50%  of  the  variance.  The  correlation 
between  the  TLX  item  on  mental  demand  and  the  maximum  ATWIT  rating  was  much  smaller 
and  explained  only  25%  of  the  variance.  The  ATWIT  ratings  showed  a  trend  similar  to  the  TLX 
ratings.  The  maximum  ATWIT  rating  displayed  an  interaction  between  the  effect  of  increasing 
task  load  and  introducing  visual  noise.  The  ATWIT  device  required  the  ATCS  to  enter  a 
subjective  workload  rating  every  5  minutes.  The  amount  of  time  required  responding  to  the 
ATWIT  device  was  minimal  as  reflected  by  the  oculometer  measurements.  On  average,  ATCSs 
spent  less  than  1.5  seconds  per  5-niinute  interval  fixating  on  the  ATWIT  device. 

One  item  in  the  Post-Scenario  Questionnaire  asked  controllers  to  rate  workload  on  that  run.  The 
effects  of  visual  noise  and  task  load  were  not  additive.  The  presence  of  visual  noise  influenced 
perceived  workload.  This  is  a  subtle  effect,  possibly  related  to  the  way  controllers  filter 
information.  With  visual  noise  present,  the  filters  are  active,  and  the  workload  does  not  seem  as 
intense.  When  visual  noise  was  present,  die  ATCSs  perceived  that  they  worked  harder  under  low 
task  load  conditions  but  were  not  working  as  hard  under  high  task  load  conditions. 

The  simulations  used  in  this  experiment  included  high  altitude  overflights  as  visual  noise.  The 
presentation  of  the  visual  noise  was  a  close  replication  of  the  traffic  normally  seen  over  the 
airspace.  Therefore,  ATCSs  may  have  developed  efficient  filtering  mechanisms  to  distinguish 
between  aircraft  within  and  outside  their  airspace.  During  the  site  visits  to  the  TRACON, 

ATCSs  indicated  that  they  filtered  out  the  representations  of  high  altitude  aircraft.  In  a 
TRACON  level  3  airspace,  VFR  aircraft  may  enter  the  airspace  represented  on  the  radarscope  in 
an  identical  fashion  as  the  high  altitude  aircraft.  When  asked  how  they  distinguished  between 
VFR  aircraft  within  the  airspace  and  the  high  altitude  aircraft,  ATCSs  responded  that  they 
compare  speeds.  This  indicates  that  controllers  do  observe  the  high  altitude  aircraft.  If  that  were 
the  case,  the  presence  of  visual  noise  would  increase  the  demand  on  cognitive  resources.  The 
workload  measures  used  in  the  current  experiment  do  not  support  this.  There  is  no  reported 
increase  in  workload  with  the  introduction  of  visual  noise.  TMs  filtering  is  undoubtedly  a 
subattentive  cognitive  process  that  experienced  controllers  develop  so  that  they  can  make  optimal 
use  of  limited  attentional  resources. 

4.3  The  Effect  of  hicrea.sing  Task  Load  and  Visual  Noise  on  Situation  Awareness  Measures 

When  task  load  increased,  ATCSs  perceived  that  their  SA  decreased.  This  is  true  for  general  SA, 
SA  for  current  and  projected  aircraft  locations,  and  SA  for  potential  conflicts.  Introducing  visual 
noise  increased  the  perceived  SA  for  potential  conflicts  sli^tly  but  significantly.  These  are 
controllers’ perceptions  that  may  not  accurately  reflect  what  they  have  cq)tured  in  working 
memory. 
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How  well  does  this  correspond  with  the  OTS  rater’s  observations?  The  OTS  rater  did  not 
observe  an  effect  of  introducing  visual  noise  on  ATCSs’  SA.  The  OTS  rater  observed  that 
maintaining  awareness  of  aircraft  position  was  lower  under  high  task  load.  The  OTS  rater’s 
observation  corresponded  well  with  ATCSs’  own  perception  of  an  SA  decrease  for  current  and 
projected  aircraft  positions.  The  OTS  rater  observed  a  decreased  ability  to  detect  pilot 
deviatioiis,  to  correct  their  own  errors,  and  to  maintain  separation.  These  observations 
corresponded  well  with  ATCSs’  own  perception  of  decreased  SA  for  potential  conflicts.  The  fact 
that  the  OTS  rater  was  aware  that  the  visual  noise  did  not  interfere  with  air  traffic  in  the  sector 
may  explain  why  the  ATCSs’  own  perception  of  a  heightened  awareness  for  potential  conflicts 
with  introducing  visual  noise  did  not  surface  in  the  OTS  rater’s  observations. 

Asking  even  an  experienced  ATCS  to  estimate  the  SA  of  someone  else  is  admittedly  asking  a  lot. 
Observer  expectations  and  biases  have  to  play  a  role.  These  data  are  suggestive,  at  best.  Only 
the  operating  controllers  really  knows  what  they  are  thinking,  and  experience  and  other  factors 
filter  even  that. 

In  the  presence  of  visual  noise,  the  radarscope  contains  many  more  aircraft  representations  than 
Avithout  visual  noise.  In  the  field,  the  radarscope  contains  the  visual  noise  as  well.  The  task 
environment  with  visual  noise  is  closer  to  ATCS  reality  than  one  without  it.  The  processing 
strategies  used  by  ATCSs  to  separate  aircraft  may  include  or  even  depend,  to  some  extent,  on  the 
presence  of  the  high  altitude  aircraft  representations.  ATCSs  are  experts  in  the  task  they 

perform.  Expertise  is  very  susceptible  to  small  changes  in  the  task  environment.  The 

participants  in  this  study  were  active  ATCSs  for  many  years.  For  them,  the  absence  of  visual 
noise  may  be  more  out  of  the  ordinary  than  the  situation  with  visual  noise  and  could  explain  the 
ATCSs’  perception  of  a  better  awareness  for  potential  conflicts. 

4.4  The  Effect  of  Task  T^ad  and  Visual  Noise  on  Eve  Movements 

ATCSs  are  supervisory  controllers,  that  is,  they  indirectly  act  upon  the  equipment  that  is  under 
their  control.  Pilots  are,  in  this  respect,  the  human  actuators  that  implement  the  ATCS 
instructions.  Compared  with  operators  of  other  equipment,  the  ATCSs  have  additional 
challenges.  The  objects  on  their  display,  unlike  other  operational  environments,  are  not 
stationary  but  move  across  the  radarscope.  The  location  of  the  radar  return  represents  the  aircraft 
position  at  one  point  in  time  in  the  airspace,  and  the  relative  movement  and  history  trails 
represent  the  heading  of  the  aircraft.  The  data  block  itself  contains  four  additional  variables: 
aircraft  call  sign,  altitude,  speed,  and  model.  ATCSs  sample  these  variables  contmuously  to 
update  their  understanding  of  the  current  state  of  the  airspace. 

The  visual  system  uses  fixations  to  retrieve  information.  During  saccades,  the  visual  system 
moves  the  eyes  but  does  not  retrieve  additional  information.  The  participants  spent  78%  of  the 
time  in  fixations.  Researchers  calculated  two  percentages  describing  fixations  broken  down  by 
scene  plane:  the  percentage  of  the  total  time  and  the  percentage  of  the  fixation  time.  The  total 
time  is  the  actual  time  available  in  a  S-minute  interval  (i.e.,  300  seconds).  The  fixation  time  is 
the  total  time  spent  in  fixations  (i.e.,  on  average,  235  seconds).  The  percentage  of  the  fixation 
time  is  a  good  indication  of  the  distribution  of  information  retrieval  across  the  scene  planes.  The 
average  duration  of  fixations  is  similar  to  those  reported  elsewhere  (Fitts,  Jones,  &  Milton,  1950; 
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Stein,  1992).  Average  saccade  durations  are  comparable  to  other  sources  as  well.  Given  these 
data,  the  eyes  are  moving  and  not  picking  up  any  viable  information  22%  of  the  total  time. 

The  literature  suggests  that  longer  fixation  durations  are  due  to  the  processing  time  necessary  for 
interpretation  of  the  information  presented  within  the  field  of  view  and  the  programming  time 
necessary  to  plan  the  next  saccade.  Careful  interpretation  of  the  current  results  suggests  that 
ATCSs  performed  more  cognitive  processing  during  fixations  on  the  ATWIT  device  and  the 
radarscope  than  on  the  keyboard  area  and  flight  strip  bay.  When  the  ATWIT  device  prompted 
the  ATCS  to  rate  the  current  workload,  it  seemed  to  require  considerable  cognitive  processing  to 
interpret  the  10-point  scale  and  compare  the  current  workload  to  that  scale.  Alternatively,  the 
ATMT  device  is  both  a  display  and  an  input  device.  Once  ATCSs  determine  the  perceived 
workload  level,  they  enter  that  level  by  touching  the  number  on  the  ATWIT  device.  The 
fixations  to  guide  the  hand  to  the  correct  number  on  the  device  may  be  quite  long.  Researchers 
interpreted  the  longer  fixation  durations  on  the  radarscope  and  aircraft  in  a  similar  fashion. 
Considerably  more  cognitive  processing  takes  place  during  fixations  on  aircraft  than  on  any  other 
radarscope  object.  The  fixation  durations  on  aircraft  correspond  well  with  durations  foxmd  on 
coclq)it  instruments  (Fitts  et  al.,  1950),  meter  monitoring  (Senders,  Elkind,  Grignetti  & 
Smallwood,  1964)  and  radar  watching  (Moray,  Neil,  &  Brophy,  1983)  (see  Figure  43).  The 
relatively  low  mean  fixation  duration  on  TRACON  radar  in  the  study  by  Stein  (1992)  may  be 
because  the  researchers  made  no  distinction  between  objects  at  which  the  ATCSs  looked.  In  this 
study,  the  fixations  on  aircraft  had  by  far  the  longest  durations.  Inclusion  of  other  objects  and 
scene  planes  would  drastically  reduce  the  average  dmation  of  the  fixations. 

When  divided  by  scene  plane,  a  difference  in  fixation  durations  was  apparent.  Fixations  on  the 
radarscope  average  620  ms  and  were  similar  in  duration  for  fixations  on  the  ATWIT  device.  The 
number  of  fixations  on  the  ATWIT  device  was  very  few,  as  expected.  Fixations  on  the  flight 
strip  bay  and  the  keyboard  area  were  much  shorter  in  durations  (320  and  450  ms,  respectively). 

The  human  visual  system  only  acquires  information  dxuing  fixations.  ATCSs  spent  75%  of  the 
total  fixation  time  on  the  rad^scope  and  69%  of  the  fixation  time  on  aircraft  representations. 
ATCSs  tend  to  focus  on  aircraft  rather  than  static  objects  such  as  airports,  VORs,  and 
intersections.  The  effects  of  increasing  task  load  and  introducing  visual  noise  and  the  number  of 
fixations  on  the  radarscope  interacted.  For  hi^  task  load  conditions,  the  number  of  fixations  on 
the  radarscope  was  lower  than  for  low  task  load  conditions.  Introducing  visual  noise  changed  the 
number  offixations  on  the  radarscope.  The  total  number  of  fixations  did  not  change 
significantly.  The  reduction  in  the  number  of  fixations  on  the  radarscope  resulted  in  an  increase 
in  fixations  on  odier  scene  planes.  The  finding  of  decreased  fixations  on  the  radarscope  when 
increasing  task  load  is  contrary  to  the  idea  that  human  observers  would  fiU  in  redimdant  fixations 
with  a  reduction  of  the  number  of  targets.  If  a  difference  would  occur,  one  pointing  towards  an 
increase  in  fixations  would  have  been  more  plausible.  Under  hi^  task  load,  this  situation 
presents  an  ATCS  with  more  potential  targets. 
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Figure  43.  Means  and  SDs  of  fixation  duration  in  other  studies. 

Researchers  postulate  that  the  reduction  in  fixations  on  the  radarscope  resulted  firom  ^CSs 
spending  more  time  on  flight  strip  maintenance  under  high  task  load  conditions.  In  a 
environment,  ATCSs  move  active  flight  progress  strips  to  the  console  and  create  new  flight 
progress  strips  for  incoming  VFR  aircraft.  The  data  suggest  that  increasing  task  load  diverts 
some  of  the  ATCS’s  attention  to  tiiese  tasks  resulting  in  fewer  fixations  on  the  radarscope. 
Indeed,  for  high  task  load  conditions  in  the  presence  of  visual  noise,  boA  the  nimb^  of  feations 
on  the  flight  strip  bay  and  the  number  on  the  keyboard  area  increased.  The  fixation  duration  of 
fixations  on  the  flight  strip  bay  decreases  as  a  result  of  an  increase  in  task  load. 

At  the  most  detailed  level,  this  study  distinguished  between  fixations  on  °n 

radarscope.  The  average  duration  of  fixations  on  aircraft  stood  out  mark^y  with  660  ms. 
is  a  relatively  long  fixation  allowing  less  than  two  stops  per  second  to  gather  mformation.  It 
suggests  considerable  cognitive  processing  by  the  ATCS.  To 

piple  in  everyday  activities  probably  scan  3  to  5  times  per  second.  O&er  objects  on  the  radar 

display  had  fixations  that  ranged  on  average  from  30  ms  to  400  ms  The 

the^preview  area  decreased  with  an  increase  in  task  load.  With  higher  task  load  the 

ATCS  spends  less  time  verifying  the  correctness  of  the  data  entoed  through  &e 

although  the  keyboard  data  indicate  that  ATCSs  type  in  more  information  under  hi^  task  load 

conditions.  ATCSs  seem  to  become  more  tactical  and  less  strategic  as  time  demands  impinge 

due  to  higher  task  load.  The  visual  scanning  data  appear  to  document  what  was  anecdotal  m  the 

past. 


41 


On  average,  the  number  of  fixations  on  the  radarscope  is  about  1  per  second.  With  an  update  rate 
of  the  radar  of  4.5  seconds,  that  allows  a  controller  to  scan  the  present  situation  in  four  fixations. 
Unless  the  controller  has  found  a  way  to  get  around  working  memory  limitations,  this  would  not 
allow  him  to  keep  the  “picture”  up  to  date.  Even  if  the  only  thing  that  changes  is  the  aircraft 
position,  this  will  introduce  uncertainty  into  the  controller’s  awareness  of  tiie  current  state  of  the 
system.  In  a  TRACON  environment,  many  aircraft  are  on  a  climbing  or  descending  course, 
which  increases  the  level  of  uncertainty  the  controller  must  take  into  account  when  making 
decisions.  All  this  becomes  even  more  remarkable  when  we  take  into  account  Moray’s  comment 
on  forgetting  (Moray  et  al.,  1983).  He  suggests  that  forgetting  sampled  material  stored  in 
working  memory  takes  place  after  12  to  15  seconds.  Therefore,  if  the  controller  uses  working 
memory,  by  the  time  the  controller  has  updated  the  current  state  of,  at  most,  12  aircraft,  his 
imcertainty  increases,  not  only  because  of  the  change  in  the  state  of  the  aircraft  but  because  of 
memory  decay  as  well. 

The  approach  used  in  this  study  to  analyze  the  effects  of  intrusions  compared  5-minute  intervals 
between  replication  scenarios.  For  each  task  load/visual  noise  combination,  two  simulations 
existed.  The  analysis  consisted  of  a  comparison  of  eye  movement  characteristics  between  a  5- 
minute  interval  that  contained  an  intrusion  with  that  same  interval  in  the  simulation  that 
replicated  the  task  load/visual  noise  combination.  The  analysis  of  the  general  eye  movement 
characteristics  did  not  show  an  effect  of  intrusions.  Several  explanations  of  the  lack  of  eye 
movement  characteristic  changes  exist.  First,  the  approach  of  using  5-minute  intervals  may  be  a 
window  of  time  that  is  too  wide  to  detect  an  effect  of  an  intruder.  Alternative  analysis  methods 
maybe  necessary  to  detect  short-term  (less  than  5  minutes)  effects  of  intruders  on  general  eye 
movement  characteristics.  Second,  the  current  approach  assumes  that  the  intruder  detection  takes 
place  at  the  time  the  aircraft  first  becomes  an  intruder.  The  current  study  did  not  include  a 
procedure  to  track  actual  intruder  detection  times.  Comments  by  ATCSs  suggest  there  are  more 
than  5  minutes  between  the  introduction  of  an  intruder  and  the  time  of  actual  detection.  Some  of 
the  ATCSs  exclaimed  “where  did  he  come  from!”  after  an  aircraft  flew  through  Class  C  airspace 
and  subsequently  was  on  its  way  out  of  the  airspace.  This  can  result  in  the  effect  of  intruder 
detection  to  occur  in  a  5-minute  interval  other  than  the  one  where  the  intrusion  initially  occurred. 
Finally,  the  research  team  went  out  of  its  way  to  present  the  ATCSs  with  a  simulated  airspace 
closely  resembling  their  actual  airspace.  The  VFR  aircraft  that  entered  Class  C  airspace  as 
intruders  represented  business  as  usual.  If  the  ATCS  should  see  business  as  usual,  one  would  not 
expect  a  change  in  general  eye  movements.  Also,  ATCSs  are  experts  in  the  sense  that  they  have 
developed  highly  automated  cognitive  processes  to  digest  large  amounts  of  data.  The  cognitive 
part  of  the  visual  system  in  case  of  highly  automated  processes  can  drive  perception.  This  would 
lead  the  ATCS  to  not  see  or  perceive  unexpected  items  or  situations.  The  IFR  intrusions  in  the 
current  study  “fell”  into  Class  C  airspace,  an  event  that  occurs  very  infrequently.  The  visual 
system’s  automaticity  may  prevent  the  ATCS  from  noticing  the  anomaly,  resulting  in  general  eye 
movements  that  do  not  show  an  effect  of  the  introduction  of  Class  C  incursions. 

Although  the  analysis  of  intervals  that  contained  incursions  into  Class  C  airspace  did  not  reveal  a 
difference  in  eye  movement  characteristics,  the  comments  by  the  OTS  rater  clearly  showed  that 
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some  of  the  controllers  did  not  detect  one  or  both  of  the  Class  C  airspace  violations.  This  was 
especially  frequent  for  scenarios  with  high  task  load  and  visual  noise  conditions.  The  OTS  rater 
indicated  that  under  baseline  condition  (i.e.,  low  task  load,  no  visual  noise)  present,  90%  of  the 
controllers  observed  both  intruders.  Under  worst  case  conditions  (i.e.,  hi^  task  load,  visual 
noise  present)  only  20%  of  the  controllers  indicated  that  they  had  observed  both  intruders. 

5.  Conclusions 

Increasing  task  load  led  to  a  larger  area  covered  per  fixation,  a  decreased  number  of  fixations  on 
the  radarscope,  and  more  fixations  on  the  flight  strip  bay.  The  effects  of  task  load  and  visual 
noise  on  ATCSs  visual  scanning  characteristics  are  often  complex.  When  task  load  and  visual 
noise  do  not  interact,  they  sometimes  produce  additive  effects. 

Scanning  behavior  is  much  more  complex  than  solely  looking  at  information  displays. 
Environmental  context  has  a  critical  impact.  Past  ATC  experience  likely  influences  ATCS 
decision  rules  on  how  and  where  to  apportion  the  limited  attentional  resources  and  will  temper 
the  visual  scanning  strategies. 

Visual  noise  and  task  load  affect  fixations  related  to  radarscope  objects  and  scene  planes  more 
than  general  eye  movements.  It  seems  that  a  relevant  metric  to  capture  visual  scanning 
characteristics  should  relate  eye  movements  to  operationally  relevant  information. 

This  research  provides  greater  understanding  of  how  ATCSs  use  current  information  displays. 
The  research  results  have  potential  for  increasing  future  ATCS  efficiency  through  improved 
display  technology  or  application  of  new  training  techniques. 
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Acronyms 


ANOVA 

Analysis  of  Variance 

API 

Aircraft  Proximity  Index 

ASL 

Applied  Science  Laboratories 

ATC 

Air  Traffic  Control 

ATCS 

Air  Traffic  Control  Specialist 

ATWIT 

Air  Traffic  Workload  Input  Technique 

DR&A 

Data  Reduction  Sc  Analysis 

DV 

Dependent  Variable 

EOS2 

Experiment  Observation  Room  2 

ER2 

Experiment  Room  2 

ER4 

Experiment  Room  4 

FAA 

Federal  Aviation  Administration 

FPL 

Full  Performance  Level 

IFR 

Instrument  Flight  Rules 

ILS 

Instrument  Landing  System 

IV 

Independent  Variable 

LED 

Light  Emitting  Diode 

LOA 

Letter  of  Agreement 

MANOVA 

Multiple  Analysis  of  Variance 

MHT 

Magnetic  Head  Tracker 

NASA 

National  Aeronautical  and  Space  Administration 

OTS 

Over-the-Shoulder 

POG 

Point  of  Gaze 

RDHFL 

Research  Development  and  Human  Factors  Laboratory 

SA 

Situation  Awareness 

SD 

Standard  Deviation 

SME 

Subject  Matter  Expert 

SOP 

Standard  Operating  Procedure 

SWAT 

Subjective  Workload  Assessment  Technique 

TLX 

Task  Load  Index 

TRACON 

Terminal  Radar  Approach  Control 

VFR 

Visual  Flight  Rules 

VOR 

VHF  Omni-directional  Range 
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Appendix  A 
Equipment  Description 


Console  Configuration 

The  experiment  consisted  of  one  ATCS  station  equipped  with  a  radarscope,  fiill  flight  strip  bay, 
an  ARTS  HI  keyboard,  and  a  trackball.  The  radarscope  ran  on  a  2,000  by  2,000  pixel  video 
display  unit. 

Simulation  Pilot  Terminal  Configuration 

A  network  permitted  chaining  of  two  simulation  operator  displays.  Researchers  saved  all  data 
into  a  directory  named  uniquely  for  each  ATCS  (ATCS  code,  data  source,  and  scenario  run). 

Each  simulation  pilot  station,  configured  for  the  simulation  pilots,  allowed  entry  of  simulation 
pilot  and  ghost  ATCS  commands.  A  secondary  radar  representation  allowed  the  readback 
position  to  track  aircraft.  The  terminals  located  in  ER2  were  sound  proofed  firom  ER4. 

Video  Camera  and  Video  Tape  Configuration 

Researchers  taped  the  video  images  of  both  the  ATCS  and  a  replication  of  the  Plan  View 
Display.  The  ATCS  position  and  flight  strip  bay  were  video  taped  using  a  low  li^t,  black  and 
white  camera.  The  video  monitors  in  EOS2  provided  a  video  display  of  all  experiment  rooms 
and  computer  screens  to  the  experimenter. 

Communications  Configuration 

Researchers  set  up  communication  links  between  the  ATCS,  OTS  observer,  simulation  pilots, 

and  experimenters.  The  equipment  monitored  communications  and  recorded  times  and 

frequencies  for  subsequent  submission  to  the  Data  Reduction  and  Analysis  (DR&A)  module. 


Oculometer 

The  ASL  eye  tracking  system  consists  of  a  headband  with  a  camera,  optics  system,  a  visor,  a 
scene  camera  assembly,  a  camera  control  unit,  an  eye  tracking  system  control  umt,  a  personal 
computer  with  interface  cards,  and  software. 

Headband  Assembly 

The  headband  assembly  is  an  adjustable  headband  with  an  optics  module  and  a  clear  plastic  visor 

plate.  The  optics  module  contains  an  eye  camera  and  illuminator.  The  illuminator  creates  a  near 

infrared  beam.  The  researchers  aim  one  part  of  the  beam  at  the  left  half  of  the  ^sor  mounted  m 
front  of  the  viewer’s  eye.  The  left  half  of  the  visor  has  a  coating  that  is  very  reflective  m  the  ne^ 
infrared  range  and  transmissive  in  the  visible  spectrum.  The  visor  deflects  the  beam  mto  the  left 
eye  of  the  viewer,  illuminating  the  viewer’s  pupil  and  cornea.  An  eye  camera  connected  to  a 
camera  control  unit  collects  the  image  reflected  by  the  visor.  The  scene  cam^  projndes  a 
reference  frame  for  line  of  gaze  positioning.  This  camera  mounts  either  on  the  headband  or  on  a 
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stationary  object.  The  control  unit  feeds  the  outgoing  signal  of  both  the  eye  and  scene  cameras 
into  the  eye  tracker  control  unit. 

Safety 

The  safe  level  of  an  oculometer  Li^t  Emitting  Diode  (LED)  is  10  mW/cm^.  ASL  (Borah,  May 
1996,  personal  communication)  testing  found  that  the  highest  radiance  level  that  the  LED 
delivers  to  the  plane  of  the  eye  is  0.8  mW/cm^.  Under  normal  conditions,  ASL  estimates  the 
LED  radiance  level  to  be  between  0.1  and  0.3  mW/cm^,  or  more  than  a  factor  of  30  lower  than 
the  safe  level  (J.  Borah,  personal  communication,  March  1 1,  1996). 

Eye  Tracker  Control  Unit 

The  eye  tracker  control  unit  (Series  4000)  houses  an  electronics  unit,  three  video  monitors,  a 
control  and  connector  panel,  and  power  supplies.  The  control  unit,  through  an  interface  with  a 
PC,  uses  the  eye  tracker  signal  to  gain  the  elements  of  interest,  i.e.,  the  pupil  and  corneal 
reflection  outlines  of  Ihe  viewer’s  eye.  The  unit  translates  the  data  into  pupil  diameter  and  line  of 
gaze  information  then  stores  the  data  into  data  files.  One  of  the  control  unit  monitors  displaj^ 
the  pupil  and  comeal  reflection  outlines  while  another  camera  displays  the  image  from  the  scene 
camera. 

Hardware 

A  Magnetic  Head  Tracker  (MHT)  provided  head  position  and  orientation  determined  in  six 
degrees  of  freedom.  This  option  allows  for  the  integration  of  eye  and  head  position  to  determine 
the  POG  of  the  user  in  world  coordinates.  The  MHT  hardware  is  an  Ascension  Technology 
magnetic  tracking  system  that  consists  of  a  control  box  and  a  source  and  sensor  module.  TTie 
source  module  transmits  a  magnetic  field  picked  up  by  the  sensor  module  mounted  on  the 
headband. 
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Appendix  B 
Detailed  Flight  Plans 


Type:  Call  in  Time:  Initial  Aircraft  Call  Beacon  Aircraft  Altitude  Altitude  Speed  Flight  Plan 

Depa^e  Includes  Controlling  Sign  Code  Type  (Initial)  (Requested) 

Arrival  VFR  Sector 
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Appendix  C 
Questionnaires 


Entiy  Questionnaire 


1  What  is  your  age  in  years?  years 

2.  Are  you  wearing  corrective  lenses  during  this  experiment?  DYes  DNo 

Hrtw  many  years  have  you  activelv  controlled  traffic?  years 

4  How  many  years  have  you  controlled  traffic  at  the  Atlantic  City  TRACON?  years 

s  Hrtw  many  months  in  the  past  year  have  von  activelv  controlled  traffic?  .  months 

6.  What  is  your  current  position  as  an  air  traffic  controller?  □  Developmental  □  Full  □  Other: 

Performance 

Level 

7.  Please  circle  the  number  that  best  describes  your  not  skilled  123456789  10  exfremely 

current  skill  as  an  air  traffic  controller.  skilled 

Comments:  _ _ _ _ _ _ _ — - 

8.  Please  circle  the  number  that  best  describes  the  level  of  no  stress  123456789  10  extremely 

stress  you  have  experienced  during  the  last  several  ^8^  level  of 

months 

Comments: _ _ _ _ _ _ _ _ 

9.  Please  circle  the  number  that  best  describes  your  not  1  2  3  4  5  6  7  8  9  10  extremely 

motivation  to  participate  in  this  study.  motivated  motivated 

Comments:  _ _ _ - _ _ — - - - - 

10  Please  circle  tbe  number  that  best  describes  your  stlte  not  healthy  123456789  10  extKimly 

of  health 

Comments:  _ _ _ _ _ - _ _ _ ^ - 

_ _ _ — - T- - - - - - 1 

11.  Do  you  search  the  PVD  in  one  special  way  for 

information?  If  it  depends  on  certain  fiictors,  vdiat  are 
they? 

Comments:  .  _ _ _ 


15.  Please  circle  the  number  that  best  describes  your  not  1  23  45  67  8  910  extremely 

experience  with  video  games.  experienced  experienced 

Comments:  _ _ _ _ 


Please  circle  the  number  that  best  describes  the 
ortance  of  the  following  aircraft  information:. 


16.  Aircraft  Call  Sign 


17.  Aircraft  Type 


18.  Aircraft  Beacon  Code 


19.  Controller  Ownership 


20.  Entry  Altitude 


21.  Entry  Airspeed 

22.  Entry  Fix 


23.  Exit  Altitude 


24.  Exit  Airspeed 


25.  Exit  Fix 


26.  Arrival  Airport  (within  sector) 


27.  Departure  Airport  (within  sector) 


28.  Current  Altitude 


extremely  123456789  10 
_ low _ 


29.  Ciurent  Airspeed 

30.  Current  Heading 


31.  Current  Aircraft  Location 


32.  Most  Recently  Assigned  Altitude 

33 .  Most  Recently  Assigned  Airspeed 

34.  Most  Recentiy  Assigned  Heading 

35.  Aircraft  Holding/Spinning 


36.  Aircraft  Waiting  for  Hand-ofDRelease 


37.  Aircraft  Near  Exit  Fix/Airival  Airport 

38.  Density  of  Aircraft  on  Radar  Display 


extremely  123456789  10 
low _ 


extremely  123456789  10 
_ low  _ 


extremely  123456789  10 
_ low  _ 


extremely  123456789  10 

_ low  _ 

extremely  123456789  10 

low  _ 

extremely  123456789  10 

_ low _ _ 

extremely  123456789  10 
_ low _ 

extremely  1234567  89  10 

_ jow _ _ _ 

extremely  123456789  10 
_ low _ 

extremely  123456789  10 

_ low  _ 

extremely  123456789  10 

_ low _ _ _ 

extremely  123456789  10 

_ low  _ 

extremely  123456789  10 

_ low _  .  _ 

extremely  1  2  3  4  5  6  7  8  9  10 

_ low  _ _ _ 

extremely  123456789  10 

_ low _ _ _ 

extremely  123456789  10 

_ low _ _ _ 

extremely  123456789  10 

_ low _ _ _ _ 

extremely  123456789  10 

_ low  _ _ _ 

extremely  123456789  10 

_ low  _ _ _ 

extremely  123456789  10 

_ low  _ _ _ 

extremely  123456789  10 


extremely  123456789  10 
low _ ■ 


123456789  10 


123456789  10 


123456789  10 


123456789  10 


123456789  10 


123456789  10 


123456789  10 


1234567  89  10 


123456789  10 


123456789  10 


123456789  10 


123456789  10 


123456789  10 
1  2  3  4  5  6  7  8  9  10 


123456789  10 


123456789  10 
123456789  10 
123456789  10 


123456789  10 


123456789  10 


extremely 

high 


extremely 

high 


extremely 

high 


extremely 

high 


extremely 

high 


extremely 

high 

extremely 

high 

extremely 

high 

extremely 

high 

extremely 

high 

extremely 

high 

extremely 

high 

extremely 

high 

extremely 

high 

extremely 

high 

extremely 

high 

extremely 

high  _ 

extremely 

high 

extremely 

-hi^^ _ 

extremely 

high 

extremely 

high 

extremely 

high 

extremely 


Please  circle  the  number  that  best  describes  the  importance 

of  the  following  radar  display  information:. _ 

39.  Range  Rings 


40.  System  Clock 


41.  VORs 


42.  Fixes 


43.  Airports 


44.  Restricted  Area  Boimdaries 


45.  n.S  Approaches 


46.  ILS  Outer  Marker 


47.  Runways 


48.  Holding  Patterns 


49.  Obstructions 


50.  Sector  Boimdaries 


51.  Filter  Settings 


52.  Future  Aircraft  List 


extremely 

low 

extremely 

low 

extremely 

low 

extremely 

low 

extremely 

low 

extremely 

low 

extremely 

low 

extremely 

low 

extremely 

_ low 

extremely 

_ low 

extremely 

low 

extremely 

_ low 

extremely 

low 

extremely 

_ low 

extremely 

low 


123456789  10  extremely 

_  high 

123456789  10  extremely 

_ high 

123456789  10  extremely 

_ high 

123456789  10  extremely 

_ high 

123456789  10  extremely 

_ high 

123456789  10  extremely 

_ _ high 

123456  7  89  10  extremely 

_ high 

123456789  10  extremely 

_ high 

123456789  10  extremely 

_ ^ _ 

123456789  10  extremely 

_ ^ _ 

123456789  10  extremely 

_ high 

123456789  10  extremely 

_ high 

123456789  10  extremely 

_ hi^l _ 

123456789  10  extremely 

_ _ high 

123456789  10  extremely 
_ high 


53.  Collision  Alert 


Post-Scenario  Questionnaire 


ID:  Scenario;  Date: 

1.  Please  circle  the  number  that  best  describes  how  •  extremely  123456789  10  extremely 

realistic  the  simulation  was.  unrealistic  realistic 

Comments: _ _ _ _ _ _ _ _ _ 

2.  Please  circle  the  number  that  best  describes  how  not  123456789  10  extremely 

representative  the  scenario  was  of  a  typical  workday.  representative  representative 

Comments: _ _ _ ■  - - - - 

3.  Please  circle  the  number  that  best  describes  if  the  no  mterference  123456789  10  extr^e 

ATWrr  device  interfered  with  controlling  traffic.  mterference 

Comments: _ _ _ — _ _ _ _ _ — 

4.  Please  circle  the  number  that  best  describes  if  the  no  mterference  1  2  3  4  5  6  7  8  9  10  ei^e 

oculometer  interfered  with  controlling  traffic.  ^  ®  erenc 

Comments: _ _ _ _ _ _ _ _ _ _ — - - - - 

5.  Please  circle  the  number  that  best  describes  how  weU  extremely  poor  1  2  3  4  5  6  7  8  9  10  ^tiremely 

the  simulation-pilots  responded  to  your  clearances  in 

terms  of  traffic  movement  and  call-backs. 

Comments:  _ _ _ - _ _ _ “ 

6.  Do  you  have  any  other  comments  about  your 
experiences  during  the  simulation? 

C-5 


7.  Please  circle  the  number  below  that  best  describes  how 
hard  you  were  working  during  this  scenario. 
Comments: 


not  hard 


extremely 

hard 


NASA  TLX 


14.  Please  circle  the  number  that  best  describes  the  mental  extremely  123456789  10 

demand  during  fliis  scenario. _ _ _ _J2^ _ 

15.  Please  circle  the  number  that  best  describes  the  physical  extremely  123456789  10 

demand  during  this  scenario.  _ _ - - - - - — 

16.  Please  circle  the  number  that  best  describes  the  temporal  extremely  123456789  10 

demand  during  this  scenario. _ _ - - - 

17.  Please  circle  the  number  that  best  describes  your  extremely  123  4  56789  10 

performance  during  this  scenario.  _ 125L - - -  — 

18.  Please  circle  the  number  that  best  describes  your  effort  extremely  123456789  10 

during  diis  scenario. _ _ _ _ _ _ _ - - — - — - —  — 

19.  Please  circle  the  number  that  best  describes  your  level  of  extremely  123456789  10 

frustration  during  this  scenario. _ ^ — - - - - 


extremely 

high 

extremely 

high 

extremely 

high 

extremely 

_ 

extremely 

high 

extremely 

high 


Exit  Questionnaire 


7.  Was  there  anything  that  you  found  particularly  unique 
in  die  simulation  that  you  would  not  see  at  your  home 
facility? 

Comments:  _ 


9. 


Do  you  search  the  P  VD  in  one  special  way  for 
information  or  does  it  depend  on  certain  fectors  and  if 
so,  what  are  they? 

Comments: _ ^ ^ _ _ 


10.  How  do  you  decide  whefoer  or  not  to  suppress  data? 
Comments:  _ _ _ _ _ 


1 1 .  Is  there  anydiing  about  the  study  that  we  should  have 
asked  or  that  you  would  like  to  comment  about? 
Comments: _ _ _ 


C-9 


Appendix  D 
Observer  Checklist 


Instructions  for  questions  1-24 

This  form  was  designed  to  be  used  by  instructor  certified  air  traffic  control  specialist  to  evaluate 
the  effectiveness  of  controllers  working  in  simulation  environments.  Observers  will  rate  the 
effectiveness  of  controllers  in  several  different  performance  areas  using  the  scale  show  below. 
When  making  your  ratings,  pleas  try  to  use  the  entire  scale  range  as  much  as  possible.  You  are 
encouraged  to  write  down  observations  and  you  may  make  preliminary  ratings  during  the  course 
of  the  scenario.  However,  we  recommend  that  you  wait  until  the  scenario  is  finished  before 
making  youT  final  ratings.  The  observations  you  make  do  not  need  to  be  restrict^  to  the 
performance  areas  covered  in  this  form  ands  may  include  other  areas  that  you  think  are 
important.  Also,  please  write  down  any  comments  that  may  improve  this  evaluation  form.  Your 
identity  will  remain  anonymous,  so  do  not  write  your  name  on  the  form. 


Rating  [  Ldbel  Description^  .  :  ^  .  -  •  ; — - - - 

1  Controller  demonstrated  extremely  poor  judgment  in  making  control  decisions  and 

very  firequently  made  errors 

2  Controller  demonstrated  poor  judgment  in  making  some  control  decisions  and 
occasionally  made  errors 

3  Controller  make  questionable  decisions  using  poor  control  techniques  which  led  to 
restricting  the  normal  traffic  flow 

4  Controller  demonstrated  the  ability  to  keep  aircraft  separated  but  used  spacing  and 

separation  criteria  which  was  excessive 

5  Controller  demonstrated  adequate  judgment  in  making  control  decisions 

6  Controller  demonstrated  good  judgment  in  making  control  decisions  using  efficient 

control  techniques  _  i  j  •  • 

7  Controller  frequently  demonstrated  excellent  judgment  in  makmg  control  decisions 

using  extremely  good  control  techniques  ,* 

8  Controller  always  demonstrated  excellent  judgment  in  making  even  the  most  difficult 

control  decisions  while  using  outstanding  control  techniques - - - 
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Maintaining  Safe  and  Efficient  Traffic  Flow 

1.  Maintaining  Separation  and  Resolving  Potential  Conflicts  12345678 

-  using  control  instructions  that  maintain  save  aircraft  separation 

-  detecting  and  resolving  inopending  conflicts  early 

Comments:  _ 


2.  Sequencing  arrival  and  Departure  Aircraft  Efficiently  12345678 

-  using  efficient  and  orderly  spacing  techniques  for  arrival  and  departure  aircraft 

-  maintaining  safe  arrival  and  departure  intervals  that  minimize  delays 

Comments:  _ _ _ 


3.  Using  Control  Instructions  Effectively  1,2345678 

-  providing  accurate  navigational  assistance  to  pilots 

-  avoiding  clearances  that  result  in  the  need  for  additional  instructions  to  handle 
aircraft  completely 

-  avoiding  excessive  vectoring  or  over-controlling 

Comments:  _ 


4.  Overall  Safe  and  Efficient  Traffic  Flow  Scale  Rating  12345678 

Comments:  _ _ 


D-2 


Maintaining  Attention  and  Situation  Awareness 


5.  Maintaining  Awareness  of  Aircraft  Positions  12345678 

-  avoiding  fixation  on  one  area  of  the  radarscope  when  other  areas  need  attention 

-  using  scanning  patterns  that  monitor  all  aircraft  on  the  radarscope 

Comments:  _ _ _ _ _ _ _ — - - - 

6.  Ensuring  Positive  Control  12345678 

Comments:  - - - - - 

7.  Detecting  Pilot  Deviations  from  Control  Instructions  12345678 

-  ensuring  that  pilots  follow  assigned  clearances  correctly 

-  correcting  pilot  deviations  in  a  timely  manner 

-  avoiding  excessive  vectoring  or  over-controlling 

Comments:  _  ' _ _ _ - — - - - 

8.  Correcting  Own  Errors  in  a  Timely  Manner  1  2  3  4  5  6  7  8 

Comments:  _ _  .  _ _ _ _ _ 

9.  Overall  Attention  and  Situation  Awareness  Scale  Rating  1  2  3  4  5  6  /8 

Comments:  _ _ _ _ _ _ _ 
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Prioritizing 


10.  Taking  Actions  in  an  Appropriate  Order  of  Importance  12345678 

-  resolving  situations  that  need  immediate  attention  before  handling  low  priority 
tasks 

-  issuing  control  instructions  in  a  prioritized,  structured,  and  timely  manner 

Comments: 

11.  Preplanning  Control  Actions  12345678 

-  scanning  adjacent  sectors  to  plan  for  inbound  traffic 

-  studying  pending  flight  strips  in  bay 

Comments; 

12.  Handling  Control  Tasks  for  Several  Aircraft  1  2345  678 

-  shifting  control  tasks  between 

-  avoiding  delays  in  communications  while  thinking  or  planning  control  actions 

Comments: 

13.  Marking  Flight  Strips  A^iile  Performing  Other  Tasks  12345678 

-  marking  flight  strips  accurately  while  talking  or  performing  other  tasks 

-  keeping  flight  strips  current 

Comments: 

14.  Overall  Prioritizing  Scale  Rating  123456  78 

Comments; 

Providing  Control  Information 

15.  Providing  Essential  Air  Traffic  Control  Information  1  2  34  5678 

“  providing  mandatory  services  and  advisories  to  pilots  in  a  timely  manner 
-  exchanging  essential  information 

Comments: 

16.  Providing  Additional  Air  Traffic  Control  Information  12345678 

-  providing  additional  services  when  workload  is  not  a  factor 

-  exchanging  additional  information 

Comments: 

17.  Overall  Providing  Control  Information  Scale  Rating  12345678 

Comments:  . 
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Technical  Knowledge 


18.  Showing  Knowledge  ofLOAs  and  SOPs 

-  controlling  traffic  as  depicted  in  current  LOAs  and  SOPs 

-  performing  hand-off  procedures  correctly 
Comments: 


19.  Showing  Knowledge  of  Aircraft  Capabilities  and  Limitations 

-  avoiding  clearances  that  are  beyond  aircraft  performance  parameters 

-  recognizing  the  need  for  speed  restrictions  and  wake  turbulence  separation 
Comments: 


20.  Overall  Technical  Knowledge  Scale  Rating 
Comments:  _ 


Communicating 


21 .  Using  Proper  Phraseology 

-  using  words  and  phrases  specified  in  ATP  7 1 10.65 

-  using  ATP  phraseology  that  is  appropriate  for  the  situation 

-  avoiding  the  use  of  excessive  verbiage 

Comments:  _ _ _ 


22.  Communicating  Clearly  and  Efficiently 

-  speaking  at  the  proper  volume  and  rate  for  pilots  to  understand 

-  speaking  fluently  >;(diile  scanning  or  performing  other  tasks 

-  clearance  delivery  is  complete,  correct  and  timely 

-  providing  complete  information  in  each  clearance 

Comments:  _ _ _ _ _ 


23.  T  isfPining  for  Pilot  Readbacks  and  Requests 

-  correcting  pilot  readback  errors 

-  processing  requests  correctly  in  a  timely  manner 

Comments:  _ _ _ 


24.  Overall  Communicating  Scale  Rating 


Comments: 


1  2  3  4  5  6  7  8 


1  2  3  4  5  6  7  8 


1  2  3  4  5  6  7  8 


1  2  3  4  5  6  7  8 


1  2  3  4  5  6  7  8 


1  2  3  4  5  6  7  8 


12345678 
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Instractions  for  questions  25-35 


The  following  questions  have  as  scale  ranging  from  1  to  10.  Where  1  represents  “extremely 
low,”  “extremely  infrequent,”  “strongly  disagree”,  etc.  and  10  represents  the  other  extreme  of  the 
spectrum. 

These  questions  are  the  same  as  we  have  asked  the  controller  after  the  scenario.  We  would  like 
you  to  give  us  your  impression  of  how  these  questions  will  be  rated  by  the  controller. 
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Appendix  E 
Performance  Variable 


Table  E-1.  Performance  Variables 


Performance  Data  _ _ 


Conflicts: _ _ 


1.  No.  Conflicts  _ _ 


2.  Dur.  Conflicts  _ 


3.  No.  Standard  Conflicts _ _ 


4.  Standard  Conflicts  API  (Aircraft  Proximity  Index 


5.  Mean  Standard  Conflicts  API _ 


6.  Dur.  Standard  Conflicts _ 


7.  No.  Longitudinal  Conflicts _ _ 


8.  No.  Longitudinal  Conflicts  API _ 


9.  Mean  Longitudinal  Conflicts  API _ 


10.  Closest-Point-of-Approach  _ 


seconds 


seconds 


11.  Oimulative  Average  System  Activi 


12.  Altitude  Changes  _ _ 


13.  Heading  Changes _ _ 


14.  No.  Speed  Changes _ 


Error:  _ _ 


15.  No.  hand-offs  Outside  Bounda 


16.  No.  Tum/HoldDe^ 


17.  Dur.  Tum/Hold  Delays _ _ 


18.  No.  Start  Point  Delays _ _ 


19.  Dur.  Start  Point  Delays _ _ 


Communications:  ■ _ _ 


20.  No.  Ground-to- Air  Contacts _ 


21.  Dur.  Ground-To-Air  Contacts _ 


22.  No.  ATCS  Messages _ _ 


23.  No.  Pilot  Message  Key  Strokes 


Task  load:  _ 


24.  No.  Aircraft  Handled _ _ 


25.  Dur.  Aircraft  Time  Under  Control 


26.  Distance  Flown _ _ 


27.  No.  Completed  Flights _ _ 

28.  No.  Departure  Altitude  Not  Attained 


29.  No.  Arrival  Altitude  Not  Attained 


30.  No.  hand-offs  Accepted  _ _ 


31.  Hand-off  Accept  Delay  Time 


seconds 


seconds 


seconds 


seconds 


miles 


seconds 


Appendix  F 

Visual  Scanning  Variables 


Target 

Targets  are  objects,  either  stationary  of  moving  that  can  be  looked  at  by  an  ATCS  (Table  F-1) 


Table  F-1.  Visual  Scanning  Targets 


Targets 

ID  needed 

Stationary 

V 

Radar  Returns 

V 

Data  Blocks 

V 

Keyboard 

Track  Ball 

Flight  Strips 

V 

ATWIT  Panel 

When  an  ID  is  needed  that  will  mean  that  the  total  number  of  targets  includes  each  of  the  targets 
within  a  category.  Stationary  targets  are  ATCoach  fixes  like  the  VOR5,  ILS  lines,  flight  table, 
etc. 

Fixation 

A  fixation  is  a  sequence  of  at  least  6  oculometer  samples  with  an  intersample  distance  of  less 
than  1  degree  of  visual  angle.  At  1  meter  distance  this  corresponds  to  a  circle  with  a  8.73  mm 
radius.  The  distance  between  two  samples  is  the  norm  of  the  vectorial  difference  of  the  sample 
coordinates.  If  2  fixations  are  not  separated  by  either  a  blink  or  a  saccade  (see  definitions  below), 
these  fixations  should  be  combined  within  one  fixation.  La  summary: 

Fixation  if: 

D  =  V((xi-xi+i)^ +(yi-yi+i)^)  >8.73  mm 

with  D  the  distance  between  to  subsequent  samples  x  ^d 
y  the  horizontal  and  vertical  point  of  gaze  coordinates  in 
mm  respectively 

and: 

>  5  with  n  the  number  of  samples  in  a  sequence 

and 

separated  by  a  blink  or  a  saccade 

Related  to  a  fixation  the  following  variables  need  to  be  calculated:  Fixation  Duration  and 
Fixation  Area.  Fixation  Area  is  an  approximation  of  the  area  covered  by  the  POG  due  to  eye 

movements  within  a  fixation. 
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Fixation  Duration: 


FDQDUR  =  tsampie  *  ^samples 

with  tsampie  where  the  duration  of  a  sample  (Vso  second) 
and  Zsample  is  the  total  number  of  samples  within  a 
fixation 

Fixation  Area: 

FDCAREA  =  (max(xfix)-min(Xfix))*(max(yfix)-min(yfix)) 

with  Xfix  and  yfix  the  sequences  of  horizontal  and  vertical 
POG  coordinates  within  a  fixation  respectively 

Blink 

A  blink  is  the  complete  or  partial  closure  of  the  eye.  The  oculometer  will  suggest  that  the 
velocity  at  the  start  and  end  of  a  blink  was  greater  than  700  degrees  per  second  which 
corresponds  with  6.108  "*4.  This  is  physically  impossible,  but  it  does  give  us  a  way  to  determine 
start  and  end  of  a  blink.  A  blink  starts  after  the  last  sample  of  the  previous  fixation  and  stops 
before  the  first  sample  of  the  next  fixation.  Insummar)^: 

Blink  if: 

VEL  =  V((Xi-Xi+i)^+(yi-yi+l)^)/ tsampie  >  6.108  “/s 

with  VEL  being  the  a  crude  estimate  of  the  tangential 
velocity  and  x  and  y  the  horizontal  and  vertical  point  of 
gaze  coordinates  in  mm  respectively.  The  index  denotes 
die  current  sample  i  and  next  sample  i+1  respectively 

and: 

n  >12  with  n  the  number  of  samples  in  a  sequence 

Related  to  a  blink  the  following  variables  need  to  be  calculated:  Fixation  Duration  and  Blink 
Distance.  Blink  Distance  is  the  distance  covered  by  the  POG  due  to  eye  movements  during  a 
blink. 

Blink  Duration: 

BLNKDUR  =  tsampie  *  Ssamples 

with  tsampie  where  the  duration  of  a  sample  (Veo  second) 
and  Ssample  is  the  total  number  of  samples  within  a  blink 

Blink  Distance: 

BLNKDST  =  (x„-xp)*(y„-yp) 

with  X  and  y  the  horizontal  and  vertical  point  of  gaze 
coordinates  in  mm  respectively.  The  index  denotes  the 
last  sample  of  the  previous  fixation  p  and  first  sample  of 
the  n^xt  fixation  n  respectively 

Saccade 

A  saccade  is  the  ballistic  movement  of  the  eye  from  one  fixation  to  the  next.  A  saccade  is 
characterized  by  fast  eye  movements  of  up  to  700  degrees  per  second.  The  cut-off  for  a  saccade 
is  a  difference  in  distance  between  two  subsequent  saccades  that  is  greater  or  equal  to  8.73  mm, 
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lasts  at  least  3  samples  (or  a  velocity  of 0.524  “4),  and  the  velocity  is  less  or  equal  to  700  degrees 
per  second  (6.108  “4).  The  saccade  will  start  at  the  end  of  the  last  sample  of  the  previous 
fixation  and  will  end  at  the  beginning  of  the  first  sample  of  the  next  fixation.  In  summary. 

0.524  >VEL>  6.108  “4 

and: 

n  >2 

Related  to  saccades  a  number  of  variables  need  to  be  calculated:  Saccade  Duration,  Saccade 
Distance,  and  Saccade  Velocity.  The  saccade  distance  is  the  angular  distance  traveled  during  a 
saccade  in  degrees.  The  saccade  velocity  is  the  average  velocity  within  a  saccade  in  degrees  per 

second. 

Saccade  Duration: 

SACDIJR=  tsampie  *  ^samples 

with  tsampie  where  the  duration  of  a  sample  ( Veo  second) 
and  Esample  is  the  total  number  of  samples  within  a 
saccade 

Saccade  Distance: 

SACDST=  (x„-Xp)*(yn-yp) 

with  X  and  y  the  horizontal  and  vertical  point  of  gaze 
coordinates  in  mm  respectively.  The  index  denotes  the 
last  sample  of  the  previous  fixation  p  and  first  sample  of 
the  next  fixation  n  respectively 

SACVEL  =  2  (V((Xi-Xi.i)2  +(yi-yi.l)^))  /  tsampie  *  Wde 

with  tsampie  where  the  duration  of  a  sample  (Veo  second) 
and  nsaccade  is  the  number  of  samples  within  the  saccade 


Dwell 


A  dwell  is  defined  as  a  sequence  of  fixations  that  return  to  a  location  within  1  degree  of  visual 
firom  a  target  location  or  within  1  degree  of  visual  angle  if  the  POG  does  not  rest  on  a  target. 
This  way  included  in  a  dwell  are  also  moving  targets. 


Related  to  dwells  a  number  of  variables  need  to  be  calculated:  Dwell  Duration  and  Dwell  ^ea. 
Dwell  Duration  is  the  duration  between  the  start  of  the  first  sample  of  the  first  fixation  and  the 
end  of  the  last  sample  of  the  last  fixation  within  a  dwell  sequence.  Dwell  Area  is  an 
approximation  of  the  area  covered  by  the  POG  within  a  dwell. 

Dwell  Duration: 

DDUR  “  tn.fixm  "  1 


with  ti,fix  1  is  the  start  of  the  first  sample  of  the  first 
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fixation  and  tn,fix  m  is  the  end  (sample  n)  of  the  last 
fixation  (fixation  m). 

Dwell  Area: 

DAREA  =  (max(xf,x)-min(xfix))*(max(yf,x)-min(yf,x)) 

with  Xfix  and  ysx  the  sequences  of  horizontal  and  vertical 
POG  coordinates  within  a  dwell  respectively 

Visual  Efficiency 

Visual  efficiency  is  defined  as  the  proportion  of  the  total  scanning  time  that  is  spent  fixating. 
Visual  Efficiency: 

VISEFF  =  (mean(FIXDUR)  *  Nfix)  / 

(mean(FIXDUR)  ♦  +  mean(SACDUR)  *  Nsac) 

In  fact,  this  is  nothing  more  than  the  portion  of  the  time  that  the  eye  is  fixed  once  the  blinks  are 
removed: 

Visual  Efficiency: 

VISEFF  =  SFIXDUR/(2FIXDUR  +  2SACDUR) 

with  SFIXDUR  the  sum  of  the  duration  of  the  fixations, 
2SACDUR  the  sum  of  the  duration  of  the  saccades  and 
TIME  the  total  time  in  seconds. 

Eve  Motion  Workload 

Eye  Motion  Workload  is  defined  as  the  average  saccade  motion  in  degrees  by  the  number  of 
saccades,  or: 

Eye  Motion  Workload: 

EYEMWL  =  mean  (SACDST)  *  Nsac  /  TIME 

with  Nsac  the  number  of  saccades  within  the  interval 
under  study  and  TIME  the  total  time  in  seconds. 

In  fact,  this  is  nothing  more  than  the  total  distance  traveled  divided  by  the  total  the  time: 

Eye  Motion  Workload: 

EYEMWL  =  2SACDST/TIME 

with  2SACDST  the  sum  of  the  distance  of  the  saccades  in 
degrees  and  TIME  the  total  time  in  seconds. 

Pupil  Motion  Workload 

Pupil  Motion  Workload  is  defined  as  the  sum  of  the  average  pupil  diameter  within  a  fixation 
divided  by  the  total  time  within  the  interval  under  consideration. 
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Pupil  Motion  Workload 

PUPMWL  =  Sl|mean(PUPDIAM)f,xi-mean(PUPDIAM)fixi+i)||/TIME 

with  PUPDIAM  the  pupil  diameter  in  mm  based  on  a 
conversion  from  ASL  arbitrary  units  to  mm  of 0.044  mm 
per  ASL  unit.  The  index  fix  i  and  fix  i+1  denote  the  i-th 
and  the  i+lth  fixation  respectively 

It  seems  if  the  author  of  the  article  that  this  measure  was  based  on  was  after  the  “distance” 
traveled  during  an  interval.  I  is  of  course  possible  to  separate  ftie  oculometer  samples  that  do  not 
include  blinks  and  then  to  calculate  the  cumulative  sum  of  the  pupil  diameter  differences.  This 
may  be  a  more  accurate  estimate  of  pupil  workload: 

Pupil  Average  Work: 

for  fixations  or  saccades: 

PUPAW  =  S||PUPDIAMi-PUPDIAMi+i|| 

with  i  and  i+1  oculometer  sample  i  an  i+1  respectively. 

In  this  case  the  oculometer  samples  that  occur  during 
blinks  are  removed  from  the  timeseries  of  data. 


Conditional  Information 

The  conditional  information  is  defined  by  Brillouin  (1962)  as  described  in  Ellis  (1986).  The 
formula  will  here  be  given  without  getting  too  much  into  the  details: 

CONINF  =  E  pi  *  [2  pij  *  log2  (pij)]  with  i  ^  j 


with  pi  is  simple  probability  of  viewing  target  i,  and  py  is 
the  probability  of  a  transition  from  target  i  to  target). 
Simple  probability  was  defined  by  Ellis  (1986)  as  the 
percentage  of  time  spent  on  each  particular  target  or 
jumping  between  each  target.  Here  we  will  calculate  it 
not  as  a  percentage  of  time,  but  the  ratio  of  the  number  of 
times  on  a  target  and  the  total  number  of  fixations  and  the 
number  of  transitions  and  the  total  number  of  saccades 
for  pi  and  pi  j  respectively. 

The  current  experiment  used  the  selected  visual  scanning  Usted  in  Table  F-2 
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Table  F-2.  Visual  Scanning  Variables 


Visual  Scanning 

Units 

1 .  Number  of  Fixations 

2 .  Mean  Duration  of  F ixations 

seconds 

3.  Mean  Fixation  Area 

inches^ 

4.  Number  of  Blinks 

5.  Mean  Blink  Duration 

seconds 

6.  Mean  Distance  Traveled  Within  A  Blink 

inch 

7.  Mean  Duration  of  Saccades 

seconds 

8.  Mean  Distance  of  Saccades 

inch 

9.  Mean  Pupil  Diameter 

millimeter 

1 0.  Mean  Duration  of  Fixations  on  Radarscope 

seconds 

1 1 .  Mean  Duration  of  Fixations  on  Keyboard  Area 

seconds 

12.  Mean  Duration  of  Fixations  on  ATWIT  Device 

seconds 

13.  Mean  Duration  of  Fixations  on  Flight  Strip  Bay 

seconds 

14.  Mean  Duration  ofFixations  on  Aircraft 

seconds 

15.  Mean  Duration  ofFixations  on  Static  Objects 

seconds 

16.  Mean  Duration  ofFixations  on  Departure  List 

seconds 

17.  Mean  Duration  ofFixations  on  System  Settings 

seconds 

1 8 .  Mean  Duration  of  Fixations  on  Preview  Area 

seconds 

1 9.  Mean  Duration  of  Fixations  on  C  A/LA  Area 

seconds 

20.  Visual  Efficiency 

21.  Eye  Motion  Workload 

inch/second 

22.  Pupil  Motion  Workload 

millimeter/second 
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Appendix  G 
Scenarios  and  Schedule 


Table  G-1 .  Overview  of  Dates  and  Test  Events 


Date 

Eyent 

May  20 -24 

Pilot  Data  Collection  (2  Ss) 

May  27  -  June  1 

Procedure  and  Data  Screening 

June  3  -  June  28 

Final  Data  Collection  (8  Ss) 

July  1  -  July  26 

Data  Anal'v^is 

July  29  -  August  23 

Table  G-2.  Two  Day  Timeline  for  Atlantic  City  ATCS 


I  Day  1 

Time 

Eyent 

Facilities  Used 

830 

Welcome  Act’s  +  Entry  Ouestionnaire 

Briefing  Room 

900 

Sector  Briefing 

(( 

945 

Tour  Facilities 

ER4 

1015 

Coffee  Break 

“ 

1030 

Eauinment  Familiarization  Run 

ER4/EOS4/Black  Room 

1100 

Break 

- 

1130 

Experimental  Run  I 

ER4/EOS4/Black  Room 

1230 

Lunch 

- 

1330 

Experimental  Run  11 

ER4/EOS4/Black  Room 

1430 

Break 

- 

1500 

Experimental  Run  HI 

ER4/EOS4/Black  Room 

1600 

Data  Backup 

ER4/EOS4 

1  Day  2 

Time 

Eyent 

Facilities  Used 

815 

Simulation  Reyiew  (if  necessary) 

ER4 

830 

Experimental  Rim  IV 

ER4/EOS4/Black  Room 

930 

Break 

- 

1000 

Experimental  Run  V 

ER4/EOS4/Black  Room 

1100 

Break 

- 

1130 

Experimental  Run  VI 

ER4/EOS4/Black  Room 

1230 

Lunch 

- 

1330 

Experimental  Run  vn 

ER4/EOS4/Black  Room 

1430 

Break 

- 

1500 

Experimental  Run  Vni 

ER4/EOS4/Black  Room 

1600 

Exit  Ouestionnaire 

ER4 

1630 

Data  Backup 

ER4/EOS4 

1700 

End 

" 
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Table  G-3.  Idealized  Participant  Schedule  Broken  Down  by  Days 


Month 

Date 

Day 

Participant  # 

May 

20 

1 

Pilot  Participant  1 

(( 

21 

2 

Pilot  Participant  1 

(C 

22 

1 

Pilot  Participant  2 

« 

23 

2 

Pilot  Participant  2 

Break  to  check/redo  data/procedures  (5/27  is  Memorial  Day). 

June 

3 

1 

Participant  1 

tt 

4 

2 

Participant  1 

a 

5 

1 

Participant  2 

6 

2 

Participant  2 

cc 

Friday  -  used 

for  post  scenario  procedures 

10 

1 

Participant  3 

«< 

11 

2 

Participant  3 

C( 

12 

1 

Participant  4 

« 

13 

2 

Participant  4 

C( 

Friday  -  used 

for  post  scenario  procedures 

u 

17 

1 

Participant  5 

u 

18 

2 

Participant  5 

u 

19 

1 

Participant  6 

tc 

20 

2 

Participant  6 

u 

Friday  -  used 

.  for  post  scenario  procedures 

ii 

24 

1 

Participant  7 

66 

25 

2 

Participant  7 

66 

26 

1 

Participant  8 

66 

27 

2 

Participant  8 

Table  G-4.  Scenario  Number  Based  on  IV  Level 


Scenario  # 

Task  load 

Overflight 

Intrusion  Type 

1 

low 

yes 

IFR,VFR 

2 

high 

yes 

IFR,VFR 

3 

low 

no 

VFR(2) 

4 

high 

no 

VFR(2) 

5 

low 

yes 

IFR(2) 

6 

high 

yes 

IFR(2) 

7 

low 

no 

Baseline 

8 

high 

no 

Baseline 
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Researchers  counterbalanced  the  presentation  order  of  the  scenarios  (Table  G-5). 


Table  G-5.  Counterbalancing  Scheme  for  12  ATCSs 


Week 

ATCS# 

Scenarios  for 
Day  1 

Scenarios  for 
Day  2 

1 

1 

0 

1 

2 

3 

4 

5 

6 

7 

8 

1 

2 

0 

2 

3 

4 

5 

6 

7 

8 

1 

3 

3 

0 

3 

4 

5 

6 

7 

8 

1 

2 

3 

4 

0 

4 

5 

6 

7 

8 

1 

2 

3 

4 

5 

0 

5 

6 

7 

8 

1 

2 

3 

4 

4 

6 

0 

6 

7 

8 

1 

2 

3 

4 

5 

5 

7 

0 

7 

8 

1 

2 

3 

4 

5 

6 

5 

8 

0 

8 

1 

2 

3 

4 

5 

6 

7 

6 

9 

0 

1 

2 

3 

4 

5 

6 

7 

8 

6 

10 

0 

2 

3 

4 

5 

6 

7 

8 

1 

7 

11 

0 

3 

4 

5 

6 

7 

8 

1 

2 

7 

12 

0 

4 

5 

6 

7 

8 

1 

2 

3 
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Appendix  H 

Integrated  Eye  Movement  and  Simulator  Data  File  Format 

The  text  below  is  a  small  portion  (less  than  one  second)  of  a  data  file  that  integrates  the  data 
recorded  by  the  simulator  data  with  the  eye  movement  data.  The  eye  movement  related 
information  is  extracted  fi*om  point  of  gaze  information.  The  point  of  gaze  information  is 
collected  at  a  rate  of  60  samples  per  second. 


08:20:49.75498  B: 

18  -0.415  -7.230  2.440  -1.850 

8.0528 

0.3000 

08:20:49.77164  S: 

2  -7.230  -7.310  -1.850  -2.210 

0.3688 

0.0167  22.13 

08:20:49.85498  0: 

6  -7.435  -7.310  -2.595  -2.195 

0.0500 

0.0833 

D:  USA454 

165  165  883  883  1.427  6/6 

D:  DAL79  424  424  652  652  1.963 

6/6 

D:DAL918  360  360  578  578  2.138 

6/6 

The  first  line  is  an  example  of  the  format  for  information  related  to  eye  blinks: 

•  the  start  time  of  the  observation  (08:20:49.75498), 

•  die  type  of  observation  (B:,  or  a  blink), 

•  the  number  ofpointofgaze  samples  (18), 

•  start  (horizontal  :-0.415,  vertical: -7.230  inches)  and 

•  end  (horizontal:  2.440,  vertical:  -1 .850  inches)  radarscope  coordinates, 

•  the  distance  traveled  (8.0528  inches), 

•  and  the  duration  (0.3000  seconds) 

The  second  line  is  an  example  of  the  format  for  information  related  to  saccades: 

•  the  start  time  of  the  observation  (08:20:49.77164), 

•  the  type  of  observation  (S:,  or  a  saccade), 

•  the  number  of  point  of  gaze  samples  (2), 

•  start  and  end  horizontal  coordinates: -7.230  -7.310  and 

•  start  and  end  vertical  coordinates:  - 1 .850  -2.2 1 0, 

•  the  distance  traveled  (0.3688  inches), 

•  the  duration  (0.0167  seconds), 

•  the  average  velocity  (22. 1 3  inches  per  second) 

The  third  and  following  lines  form  an  example  of  the  format  for  information  related  to  fixations: 
On  the  third  line: 

•  the  start  time  of  the  observation  (08:20:49.85498), 

•  the  type  ofobservation(0:,l:,  2:,  3:,  4:,  fixations  on  scene  planes  0-4), 

•  the  number  of  point  of  gaze  samples  (6), 
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•  start  and  end  horizontal  coordinates:  -7.435  -7.3 1 0  and 

•  start  and  end  vertical  coordinates:  - -2.595  -2.195, 

•  the  area  covered  traveled  (0.0500  square  inches), 

•  the  duration  (0.0833  seconds). 

The  line  following  the  general  fixation  information  displays  the  object  that  was  closest  to  the 
center  of  the  fixation,  in  this  case,  USA454.  The  following  indented  lines  present  a  list  of 
objects  that  are  within  a  radius  of  2  inches  away  fi-om  the  center  of  the  fixation.  The  format  is  as 
follows: 

•  the  type  of  observation  (D:,  S:  Dynamic  or  Static  Objects), 

•  start  and  end  horizontal  coordinates:  165  165  in  pixels  and 

•  start  and  end  vertical  coordinates:  883  883  in  pixels, 

•  die  distance  traveled  (1.963  inches), 

•  the  number  of  samples  the  object  was  within  the  fixation  radius  (6  out  of  6  fixation  samples) 
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Appendix  I 

Snapshots  of  Fixation  Distributions  and  Simulator  Images  and  Data 
0 
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Figure  I-l .  Fixation  Distribution  during  a  45  minute  simulation  of  a  low  task  load  scenario 
without  visual  noise.  The  units  for  horizontal  and  vertical  coordinates  are  in  pixels.  The  top  left 
comer  corresponds  with  Ihe  top  left  comer  of  the  radar  scope. 
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Figure  1-2.  Fixation  Distribution  during  a  45  minute  simulation  of  low  task  load  scenario  without 
visual  noise.  The  flight  path  of  a  departure,  BTA3721  is  superimposed.  The  circles  represent 
fixations  that  were  identified  as  fixation  on  flight  BTA3721.  The  units  for  horizontal  and 
vertical  coordinates  are  in  pixels. 
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Figure  1-3.  Simulator  Image  of  a  High  Task  Load  Scenario  with  Visual  Noise  Present.  Range 
Rings  were  set  to  5  miles. 
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0“ 

Figure  1-4.  Simulator  data  on  radar  scope  object  location  and  size,  integrated  with  point  of  gaze 
information.  Small  open  circles  represent  static  objects,  medi\im  open  circles  represent  aircraft 
not  under  control,  large  open  circles  represent  aircraft  under  control,  and  solid  small  circles 
represent  point  of  gaze  data  at  approximately  15  points  per  second. 
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Appendix  J 
Statistical  Background 

This  section  provides  the  reader  with  background  information  on  the  statistical  methods  used  in 
this  report.  These  statistical  methods  are  powerful  tools  that  allow  researchers  to  determine  the 
most  probable  outcomes  of  an  experiment  based  on  limited  sample  sizes.  The  following 
paragraphs  explain  general  concepts  in  statistics,  the  methods  utilized  in  this  study,  and  some 
important  considerations  to  use  tiiem  effectively. 

The  purpose  of  any  statistical  experiment  is  to  determine  the  effect  of  certain  factors  on  one  or 
more  outcome  variables  (dependent  variable  or  DV).  An  example  of  a  DV  is  the  number  of 
altitude  changes  an  ATCS  makes.  This  DV  could  be  affected  by  the  type  of  airspace  (terminal, 
en  route,  or  oceanic),  the  number  of  aircraft  flying  through  or  to  the  sector,  or  many  other  factors. 
The  manipulated  factors  of  an  experiment  are  the  IVs  (or  IVs).  Each  manipulation  of  an  IV  (e.g., 
25  planes  or  45  planes)  forms  a  separate  experimental  condition.  Each  trial  under  a  particular 
condition  is  termed  an  observation. 

Experiments  can  include  one  or  more  IVs.  When  an  experiment  includes  more  than  one  IV, 
multiple  IVs  can  affect  the  outcome  differently.  This  is  called  an  interaction.  It  would  be 
impossible  to  study  the  effects  of  type  of  airspace  and  number  of  aircraft  independently.  When 
such  interactions  between  IVs  occur,  the  researcher  will  study  the  effect  by  holding  one  variable 
constant  while  varying  the  others.  This  is  called  testing  for  simple  effects.  In  this  way,  the 
researcher  obtains  a  picture  of  how  the  variables  interact  by  examining  the  outcome  of  each 
manipulation.  When  researchers  study  tiie  effect  of  each  IV  separately  (no  interactions),  it  is 
termed  an  analysis  of  main  effects.  Main  effects  can  only  be  studied  in  the  absence  of 
interactions. 

The  number  of  values  for  the  IVs  included  in  an  experiment  depends  on  several  practical 
considerations.  For  example,  if  a  researcher  is  studying  the  decision-making  patterns  of 
controllers  as  a  function  of  type  of  airspace,  the  values  of  the  IV,  type  of  airspace  (tower, 
TRACON,  enroute),  are  clear,  hi  other  cases  the  answer  would  depend  upon  what  type  of 
outcome  the  researchers  needed  from  the  results  of  the  experiment  as  well  as  some  practical 
considerations.  Different  values  of  IVs,  termed  levels,  can  increase  the  number  of  experiment 
conditions  and  thus  increase  the  resources  needed  to  complete  the  experiment.  One  can  certairily 
imagine  the  complexity  and  length  of  an  experiment  in  which  controllers  with  experience  ranging 
from  1  to  50  years  creating  50  incremental  levels  were  studied.  It  would  be  far  simpler  and  easier 
to  study  the  effect  of  controller  experience  by  using  only  three  categories:  Developmental,  Full 
Performance  Level  (FPL),  and  Supervisor. 

What  is  the  number  of  observations  required  for  each  test  condition?  Increasing  the  number  of 
observations  increases  the  statistical  power  of  the  experiment,  hicreased  statistical  power  means 
that  an  increased  probability  exists  that  the  outcome  of  an  experiment  wUl  likely  be  true  for  the 
entire  population.  However,  increasing  the  number  of  observations  comes  at  the  expense  of 
greater  numbers  of  participants,  more  time,  or  both.  An  efficient  experimeiital  design  should 
include  enou^  observations  for  reasonable  statistical  power  witiiout  including  unnecessary 
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observations  that  could  dramatically  increase  demands  for  resources  unless  there  was  an 
increased  need  for  power. 

With  insight  into  statistical  terminology  as  well  as  some  background  into  considerations  involved 
in  experimental  design,  it  is  now  useful  to  look  into  several  different  categories  of  experiments 
and  statistical  methods  used  to  determine  significant  outcomes.  For  simplicity,  each  of  the 
following  categories  involve  only  a  single  IV  (the  experience  level  of  controllers).  In  increasing 
level  of  complexity,  three  categories  of  experiments  will  be  examined: 

1.  Observations  on  a  single  DV  under  two  conditions  (T-test) 

2.  Observations  on  a  single  DV  under  multiple  conditions  (ANOVA) 

3.  Observations  on  multiple  DV  under  multiple  conditions  (MANOVA) 

Each  of  these  categories  is  discussed  below. 

Observations  on  a  Single  Variable  Under  Two  Conditions 

When  a  researcher  wants  to  compare  two  conditions,  the  average  of  multiple  observations  on  a 
single  variable  are  taken  under  two  conditions,  and  the  experimenter  performs  a  T-test. 

However,  an  average  value  can  often  be  misleading.  Within  a  group  of  such  observations,  some 
differences  will  exist  in  the  individual  observations  that  contributed  to  the  average.  Some 
Developmental  controllers  may  be  faster  learners  than  others  and  will  use  less  altitude  changes  in 
order  to  control  traffic.  The  average  number  of  altitude  changes  for  all  Developmental 
controllers  can  include  a  wide  range  of  values.  The  differences  between  the  individual  times  and 
the  mean  number  of  times  represents  the  variability  of  the  data.  As  the  variability  in  the  data 
increases,  die  mean  value  is  less  useful  to  the  researcher  because  many  of  the  individual  values 
are  far  from  the  mean.  Figure  J-1  illustrates  the  variability  of  data. 


♦  Set  2 
■  Setl 
Mean 


Figure  J-1.  Two  sets  of  observations  ivith  the  same  means,  but  very  different  values. 

If  a  researcher  wants  to  compare  two  samples,  the  comparison  not  only  involves  comparing  the 
averages  but  also  the  variability  within  the  observations.  For  this  reason,  the  true  mean  (the 
mean  a  researcher  would  calculate  if  he/she  sampled  the  number  of  altitude  changes  for  all 
Developmental  controllers  in  the  world)  differs  from  the  sample  mean.  A  researcher  must  ask  if 
the  difference  in  the  means  of  these  two  sets  of  observations  is  a  true  difference  or  caused  by 
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chance.  This  is  where  probability  theory  aids  the  researcher.  Statistics  can  help  the  researcher 
determine  the  probability  that  the  two  means  for  the  entire  population  (all  controllers)  are 
different  from  the  sample  (limited  number  of  controllers).  The  statistical  test  used  in  this  case  is 
the  t-test.  The  t-test  compares  two  averages  and  checks  if  the  two  averages  are  different  due  to 
chance  alone.  It  is  important  to  recognize  that  the  t-test  never  gives  the  researcher  100% 
assurance  that  the  two  means  actually  differ.  It  is  common  practice  at  accept  a  95%  assurance 
(or,  in  other  words,  a  5%  risk)  as  sufficient  guarantee. 

SUMMARY  OF  A  T-TEST:  An  experiment  includes  multiple  observations  on  a  single  variable 
under  two  conditions.  The  average  values  (means)  of  the  two  conditions  takes  variability  into 
consideration.  The  analj^es  determines  the  probability  that  the  means  differ  due  to  chance  alone. 

Example:  When  one  compares  the  number  of  altitude  changes  between  Developmental  and  FPL 
controllers  at  a  local  center,  the  comparison  involves  multiple  observations.  The  multiple 
observations  consist  of  the  number  of  altitude  changes  of  each  individual  within  the  experience 
level.  The  variable  is  the  number  of  altitude  changes.  The  conditions  include  the  two  levels  of 
experience.  Figure  J-2  is  a  graphical  display  of  this  example.  Although  it  shows  a  difference  in 
number  of  altitude  changes  between  the  two  groups,  some  individual  observations  overlap.  A  t- 
test  examines  if  this  difference  was  caused  by  chance. 


a 

A 
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Figure  J-2.  Multiple  observations  of  altitude  changes  as  a  function  of  experience  level. 
Observations  on  a  Single  Variable  Under  Multiple  Conditions 

Where  the  t-test  compared  the  averages  between  two  conditions,  the  analysis  of  variance 
(ANOVA)  compares  averages  of  a  single  variable  between  multiple  conditions  (i.e.,  the  number 
of  altitude  changes  including  Developmental,  FPL,  and  Supervisors).  An  ANOVA  tests  if  these 
averages  are  different  due  to  chance  alone.  The  basic  test  results  in  an  F  value  for  a  single  DV 
(the  number  of  altitude  changes).  The  value  of  F  ranges  from  0  to  infinity  (oo).  A  large  F  vdue 
may  indicate  that  the  IV  (experience  level)  has  a  powerful  effect  on  the  DV  (number  of  altitude 
changes)  with  less  likelihood  that  differences  between  means  occurred  by  chance.  The  sfiength 
of  association  (e.g.,  t])  or  percent  of  variance  explained  is  an  indication  of  the  difference  in  the 
strength  of  effects  between  conditions.  A  difference  between  means  is  s^nificant  if  there  is  a 
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very  hi^  probability  that  the  means  are  actually  different  (usually  greater  than  95%).  Sometimes, 
there  is  a  significant  difference  where  the  F  value  is  relatively  low.  This  indicates  that  the  IV 
does  not  have  a  very  strong  effect. 

An  ANOVA  can  show  that  there  is  a  difference  in  means  not  caused  by  chance  alone.  If  the 
ANOVA  indicated  that  the  number  of  altitude  changes  varies  with  experience  level,  are  the  mean 
number  of  altitude  changes  for  Developmental  controllers  different  than  FPL  controllers?  The 
mean  for  Developmental  controllers  differs  significantly  fi-om  those  of  FPL  but  not  significantly 
fi:om  those  for  Supervisors.  Therefore,  another  test  needs  to  compliment  the  ANOVA.  This  test 
is  called  a  post  hoc  comparison.  Researchers  will  use  post  hoc  comparisons  to  determine  which 
of  the  pairs  of  means  differ  significantly. 

SUMMARY  OF  AN  ANOVA:  The  ANOVA  compares  averages  of  a  single  DV  between 
multiple  conditions  and  tests  if  these  averages  are  different  due  to  chance  alone.  The  test  results 
in  an  F  value.  A  large  F  value  indicates  less  likelihood  and  a  small  value  indicates  increased 
likelihood  that  differences  between  means  occurred  by  chance.  A  difference  between  means  is 
significant  if  there  is  a  very  high  probability  that  the  means  are  actually  different.  A  post  hoc 
comparison  determines  which  means  differ. 

Example:  When  a  researcher  compares  the  number  of  altitude  changes  between  Developmental, 
FPL,  and  Supervisors  at  a  local  ARTCC,  the  comparison  involves  multiple  observations.  The 
multiple  observations  are  the  number  of  altitude  changes  of  each  individual  within  each  group. 
The  variable  is  the  number  of  altitude  changes.  The  conditions  are  the  three  experience  levels. 
Figure  J-3  displays  the  data  related  to  this  example.  Some  differences  in  number  of  altitude 
changes  exist  between  experience  levels,  but  there  is  overlap  between  observations  in  each 
experience  level.  An  ANOVA  would  determine  if  these  differences  are  due  to  chance  alone.  If 
the  ANOVA  indicated  that  there  is  some  difference  in  experience  levels  regarding  nximber  of 
altitude  changes,  post  hoc  comparisons  would  indicate  which  means  associated  with  which 
experience  levels  differ. 


Figure  J-3.  Multiple  observations  of  altitude  changes  as  a  function  of  experience  level. 
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Observations  on  a  Multiple  Variable  Under  Multiple  Conditions 


Where  an  ANOVA  compares  averages  between  multiple  conditions  for  a  single  variable  (a 
univariate  test),  the  multivariate  ANOVA  (MANOVA)  compares  averages  for  several  DVs 
simultaneously  and  tests  if  these  averages  are  different  due  to  chance  alone.  Suppose  that 
researchers  wanted  to  examine  number  of  altitude  changes  and  the  number  of  heading  changes. 
Also  suppose  that  the  researchers  wanted  to  test  both  of  these  measures  as  a  function  of 
experience  level.  This  presents  a  case  of  multiple  DVs  (number  of  altitude  changes  and  number 
of  heading  changes)  and  multiple  conditions  (Developmaital,  FPL  and  Supervisor).  The 
experimenters  focus  on  how  experience  level  affects  the  set  of  variables(number  of  altitude 
changes  and  number  of  heading  changes).  A  researcher  would  not  do  two  ANOVAs  for  each  of 
the  DVs  (number  of  altitude  changes  and  number  of  heading  changes)  because  misleading 
outcomes  result  from  multiple  ANOVAs.  A  MANOVA  is  more  appropriate. 

The  basic  MANOVA  test  results  in  a  value  called  WUk’s  Lambda  (A)  that  includes  the  effects 
of  more  than  one  DV  (bofri  number  of  altitude  changes  and  number  of  heading  changes).  The 
value  of  Wilk’s  Lambda  ranges  from  zero  to  one.  The  lower  the  value  of  A,  the  more  powerful 
the  effect  of  the  IV  (experience  level)  on  the  set  of  DVs  and  the  less  likely  it  is  that  the 
differences  between  means  occurred  by  chance.  Sometimes,  tiiere  is  a  significant  difference 
where  A  is  relatively  high.  This  indicates  that  the  effect  is  not  that  strong. 

After  a  significant  result  of  a  MANOVA  test,  researchers  then  conduct  ANOVA  tests  (one  for 
number  of  altitude  changes  and  one  for  number  of  heading  changes.  Figure  J-4  depicts  an 
example  of  the  steps  taken  during  a  MANOVA.  The  example  shown  in  Figure  J-4  includes  two 
DVs. 


Figure  J-4.  Example  of  the  steps  in  a  MANOVA. 
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SUMMARY  OF  A  MANOVA:  The  MANOVA  compares  averages  for  several  variables 
simultaneously  and  tests  if  these  averages  are  different  due  to  chance  alone.  The  basic 
MANOVA  results  in  a  value  called  A  that  includes  the  effects  of  more  than  one  DV.  The  lower 
the  value  of  A,  the  more  powerful  the  effect  of  the  IV  on  the  set  of  DVs  and  the  less  likely  it  is 
tha:t  the  differences  between  means  occurred  by  chance.  After  a  significant  result  of  a 
MANOVA,  which  indicates  that  at  least  two  means  are  statistically  different  for  the  sj^tem, 
researchers  then  conduct  ANOVAs, 

Example:  When  one  compares  the  number  of  altitude  changes  and  number  of  heading  changes 
between  Developmental,  FPL  and  Supervisor  at  a  local  center,  the  comparison  involves  multiple 
observations  of  two  variables.  The  multiple  observations  are  the  number  of  altitude  changes  and 
number  of  heading  changes  of  each  individual  within  the  each  experience  level.  The  DVs  are  the 
number  of  altitude  changes  and  the  number  of  heading  changes.  The  conditions  are  formed  by 
the  three  experience  levels.  Figure  J-5  displays  the  data  for  this  example.  Without  looking  at  the 
individual  variables.  One  can  see  that  the  three  experience  levels  differ.  A  MANOVA  would 
determine  if  chance  alone  caused  these  differences.  If  the  differences  are  beyond  chance  (or 
significant  in  statistical  terms),  ANOVAs  on  the  individual  variables  are  conducted. 


♦Developmental 
■  FPL 

♦  Supervisory 


No.  Heading  Changes 


Figure  J-5.  Multiple  observations  of  niimber  of  altitude  changes  and  niunber  of  heading  changes 
as  a  function  of  experience  level. 

Summary 

The  preceding  paragraphs  give  some  insist  into  die  statistical  methods  used  by  researchers. 
Statistical  methods  are  very  powerful  tools  for  the  researcher.  They  tell  the  researcher  if  the 
experimental  conditions  affect  the  dependent  measures  tested.  The  type  of  statistical  test  that  the 
researcher  uses  varies  with  the  type  of  experiment.  A  good  researcher  will  design  experiments  so 
they  can  use  these  techniques  to  the  fullest  extent. 
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Appendix  K 
Descriptive  Statistics 


Post  Scenario  Questionnaires 

After  each  scenario,  the  ATCSs  rated  several  scenario-related  items.  The  Post-Scenario 
Questionnaire  also  included  the  six  NASA  TLX  ratings.  There  were  12  participants  and  8 
simulation  scenarios  or  a  total  of  96  observations.  Equipment  failure  during  the  simulations 
caused  the  loss  of  one  observation.  The  total  number  of  observations  used  in  the  analyses  was 
therefore  95. 


Post-Scenario  Questions 

The  scenarios  were  moderately  realistic  with  a  mean  of  6.6  and  moderately  representative  of  an 
average  day  at  die  Atlantic  City  TRACON  with  a  mean  of  6.0.  The  ATWIT  device  hardly 
interfered  with  controlling  traffic  as  indicated  by  a  mean  rating  of  interference  of  1 .6.  The 
oculometer  interfered  more  with  controlling  traffic  but  still  only  moderately  with  a  mean  rating 
of  2.9.  ATCSs  rated  the  simulation  pilots’  responses  of  very  good  quality  at  an  8.9  level.  On 
average,  the  ATCSs  worked  moderately  hard  with  a  mean  of  5. 1 .  The  self-rated  quality  of 
control  was  good  at  7.6.  The  overall  SA,  for  current  aircraft  location,  projected  aircraft  location, 
and  potential  violations  were  good  wifti  means  of  7.8, 7.8, 7.7,  and  7.8,  respectively.  The 
scenarios  were  moderately  difficult  with  a  mean  of  5.2  (Table  K-1). 


Table  K-1.  Post-Scenario  Questions  (n=95) 


Variable  Label 

Mean 

SD 

Realism 

6.59 

1.89 

Representative 

6.02 

1.92 

ATWIT  Interference 

1.62 

1.48 

Oculometer  Interference 

2.93 

2.09 

Sim.  Pilot  Response 

8.87 

1.26 

Working  Hard 

5.12 

2.70 

Quality  of  Control 

7.57 

1.60 

Overall  SA 

1.15 

1.66 

Current  Act.  location  SA 

7.75 

1.76 

Projected  Act  location  SA 

7.74 

1.75 

Potential  Violations  SA 

7.81 

1.65 

Scenario  Difficulty 

5.19 

2.74 

Post-Scenario  TLX 

The  TLX  scores  (Table  K-2)  revealed  that  the  performance  and  effort  ratings  were  high  with 
means  of  7.6  and  7.3,  respectively.  The  level  of  fhistration  was  relatively  low.  Mental,  physical, 
and  temporal  demand  were  moderate  with  means  of  5.6, 4.1,  and  4.6,  respectively. 
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Table  K-2.  Post  Scenario  TLX  (n=95) 


Variable  Label 

Mean 

SD 

Mental  Demand 

5.64 

2.70 

Physical  Demand 

4.08 

2.61 

Temporal  Demand 

4.63 

2.40 

Performance 

7.56 

1.51 

Effort 

7.31 

2.12 

Frustration 

3.85 

2.72 

The  workload  levels  found  from  the  ATWIT  ratings  correlated  with  the  workload  levels  foimd  by 
the  TLX  items  in  the  Post-Scenario  Questionnaire  (Table  K-3).  Especially  the  mental  demand 
item  correlated  well  with  the  mean  ATWIT  rating. 


Table  K-3.  Correlations  Between  Mean  and  Maximum  ATWIT  Ratings  and  Post-Scenario  TLX 

Items 


ARMean 

ARMax 

Mental  Demand 

.71 

.50 

Physical  Demand 

.46 

.43 

Temporal  Demand 

.57 

.39 

Performance 

-.20 

-.22 

Effort 

.34 

.17 

Frustration 

.53 

.35 

Over-the-Shoulder  Rating  Forms 

An  ATC  SME  conducted  an  OTS  rating.  The  items  on  the  checklist  are  similar  to  the  ones  used 
in  other  studies  except  for  five  items  that  are  replications  of  items  on  the  Post-Scenario 
Questionnaires  and  ^e  six  TLX  ratings.  The  scale  on  the  comparison  and  TLX  items  is  l-IO. 

The  other  items  have  a  scale  from  1-8.  The  dichotomy  in  scaling  will  ease  the  comparison  of 
results  with  previous  studies  and  the  responses  of  the  participants. 

Over-the-Shoulder  Ratings 

The  OTS  rater  rated  overall  performance  of  the  ATCS  participants  moderately  good  at  6.2. 
Overall  traffic  flow  efficiency  was  very  good  at  7.5.  Overall  Attention  and  SA  were  good  at  7.0. 
Overall  prioritizing  skills  were  very  good  at  7.5.  Providing  air  traffic  control  information  was 
very  good  as  well  at  7.5.  The  overall  technical  knowledge  of  the  ATCS  participants  was 
excellent  at  8.0.  The  communication  skills  of  the  participants  were  good  at  7.0.  Table  K-4 
presents  a  more  detailed  breakdown  of  the  OTS  ratings. 

Over-the-Shoulder  ratines  of  selected  Post  Scenario  questions 

To  investigate  if  an  OTSR  can  observe  control  strategy  preferences  of  the  ATCSs,  the  researchers 
replicated  five  questions  from  the  Entry  Questionnaire  to  the  OTS  rating  form.  Table  K-5 
presents  the  means  and  standard  deviations  of  these  questions.  The  OTSR  perceived  the  ATCSs 
to  have  a  preference  for  vertical  separation  and  vectoring  and  much  less  for  speed  control.  The 
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OTSR  rated  the  work  level  to  be  moderate  at  5.8.  The  ATCSs’  performance  was  very  good  at 
8.7. 


Table  K-4.  General  Over-the-Shoulder  Ratings  (n=96) 


Label 

Mean 

Std 

Overall  Performance 

6.18 

1.12 

Maintaining  Traffic  Flow 

7.33 

1.11 

Sequencing  Traffic  Flow  Efficiently  • 

7.41 

1.09 

Efficient  Control  Instructions 

7.48 

0.91 

Overall  TrafiBc  Flow  Efficiency 

7.38 

1.02 

SAofAct  Positions 

7.18 

1.18 

Positive  Control 

7.03 

1.47 

Detection  of  Control  Instruction  Deviation 

7.52 

1.04 

Correcting  Own  Errors  Timely 

7.18 

1.34 

Overall  Attention  and  SA 

7.01 

1.23 

Actions  in  Order  of  Importance 

7.40 

0.96 

Preplanning  Control  Actions 

7.48 

0.92 

Handling  Control  for  Several  Aircraft. 

7.40 

1.00 

Flight  Strip  Marking 

7.51 

0.92 

Overall  Prioritizing 

7.45 

0.87 

Providing  Essential  ATC  Info. 

7.49 

0.85 

Providing  Additional  ATC  Info. 

7.51 

0.86 

Overall  ATC  Info.  Rating 

7.47 

0.85 

Knowledge  of  LOAs  and  SOPs 

7.88 

0.36 

Knowledge  of  Act  Capabilities 

7.97 

0.17 

Overall  Technical  Knowledge 

7.95 

0.27 

Proper  Phraseology 

6.86 

0.82 

Clear  and  Efficient  Communication 

7.19 

0.89 

T  istening  for  Pilot  Readbacks  and  Request 

7.40 

0.93 

Overall  Communication  Rating 

7.04 

0.65 

Table  K-5.  Over-the-Shoulder  Ratings  of  Selected  Post-Scenario  Questions 


Label 

N 

Mean 

Std 

Ptef.  for  Vertical  Separation 

96 

8.92 

1.29 

Pref.  for  Vectoring 

95 

8.24 

1.60 

Pref.  for  Speed  Control 

46 

3.59 

3.12 

Working  Hard 

96 

5.80 

2.54 

Control  Performance 

96 

8.70 

1.51 

Over-the-Shoulder  Ratines  of  TLX  items 


The  OTSR  rated  each  of  the  TLX  items  for  each  of  the  simulation  runs.  On  average  die  mental, 
physical,  and  temporal  demand  were  moderate  at  6.2, 6.0,  and  6.5,  respectively.  The 
performance  rating  on  the  TLX  was  very  good  at  8.7.  The  TLX  effort  level  was  moderate  at  6.4. 
The  level  of  frustration  was  on  average  low  at  2.8  (Table  K-6). 
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Table  K-6.  Over-the-Shoulder  Ratings  of  TLX  Items  (n=95) 


Label 

Mean 

Std 

TLX  Mental  Demand 

6.21 

2.43 

TLX  Physical  Demand 

6.04 

2.28 

TLX  Temporal  Demand 

6.52 

2.35 

TLX  Performance 

8.71 

1.31 

TLX  Effort 

6.41 

2.35 

TLX  Frustration 

2.79 

2.02 

Visual  Scanning 

Several  levels  of  data  reduction  formed  the  basis  for  the  results  presented  here.  Fixations, 
saccades,  blinks,  and  pupil  information  formed  the  basis  for  the  visual  scanning  data  set.  This 
data  set  consisted  of  the  summary  variables  of  5-minute  intervals.  This  section  on  descriptive 
statistics  presents  the  siraimary  statistics  across  these  5-minute  intervals  across  all  conditions.  In 
this  experiment,  the  researchers  distinguished  three  levels  of  detail  in  eye  movement 
characteristics.  The  &st  level  focused  on  general  eye  movement  characteristics,  without  making 
a  distinction  between  objects  or  groups  of  objects  at  which  participants  looked.  The  second  level 
focused  on  scene  planes  or  surfaces  on  which  the  ATCSs  rested  their  gaze  (radarscope,  keyboard 
area,  flight  progress  strip  bay,  and  ATWIT  device). 

General  Eve  Movement  Characteristics 

The  first  level  of  detail  included  all  eye  movement  characteristics  (fixations,  saccades,  blinks, 
and  pupil).  The  general  visual  scanning  variables  used  in  tiie  analyses  are  the  mean  values  of  a  5- 
minute  interval  (Table  K-7).  Table  K-8  presents  the  percentage  of  time  spent  on  fixations, 
saccades,  and  blinks.  Each  5-minute  interval  contained  approximately  426  fixations.  On 
average,  the  participants  spent  78%  of  the  time  in  fixations.  The  average  fixation  duration  was 
560  ms.  During  fixations,  small  eye  movements  occurred  that  resulted  in  average  area  coverage 
of  0.67  square  inch  (435  mm^).  The  participants’  eyes  moved  in  saccades  approximately  17%  of 
the  time.  The  saccades  lasted  an  average  of  120  ms.  The  mean  distance  traveled  between  two 
fixations  was  3.30  inches  (77.19  mm).  The  mean  pupil  diameter  was  5.87  mm.  On  average, 
participants  blinked  81  times  per  5-minute  interval.  Blinks  accounted  for  7%  of  the  time.  The 
mean  blink  duration  was  250  ms.  During  closure  of  the  eyelids,  the  eye  can  still  travel.  The 
distance  traveled  within  ablink  was  9.18  inches  (23.32  mm). 
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Table  K-7.  General  Visual  Scanning  Variables  (n=864) 


Variable  Label 

Mean 

SD 

Units 

Fixation  Number 

426.00 

50.00 

Fixation  Mean  Duration 

560.00 

78.00 

ms 

Fixation  Area  Mean 

0.67 

0.12 

inch^ 

Visual  Efficiency 

0.82 

0.04 

Total  Fixation  Time 

235.00 

20.00 

second 

Saccade  Number 

431.00 

55.00 

Saccade  Mean  Duration 

120.00 

19.00 

ms 

Saccade  Mean  Distance 

3.30 

0.59 

inch 

Eve  Motion  Workload 

4.76 

1.09 

inch/sec 

Total  Saccade  Time 

50.00 

10.00 

second 

Pupil  Mean  Diameter 

5.87 

1.07 

mm 

Pupil  Motion  Workload 

0.05 

0.03 

mm/sec 

Blink  Number 

81.00 

31.00 

Blink  Mean  Duration 

253.00 

132.00 

ms 

Blink  Mean  Distance 

9.18 

4.02 

inch 

Total  Blink  Time 

21.34 

15.03 

second 

Note  tiiat  the  mean  5-minute  interval  data  formed  the  basis  for  the  calculation  of  the  percentage 
of  time.  The  total  of  the  percentage  spent  on  fixations,  saccades,  and  blinks  therefore  does  not 
add  to  100  percent  due  to  inherent  rounding  error  (K-8). 


Table  K-8.  Total  Fixation,  Saccade,  and  Blink  Time  (sec.) 


Variable  Label 

Mean 

sd" 

Percent 

Total  Ffacation  Time 

234.68 

20.12 

78.23 

Total  Saccade  Time 

49.81 

10.31 

16.60 

Total  Blink  Time 

21.34 

15.03 

7.11 

Correlations 

Considering  the  correlations  between  general  eye  movement  related  variables,  the  number  of 
saccades  is  not  included  in  the  inferential  statistical  analysis.  With  a  correlation  coefficient  of 
0.99  between  the  number  of  saccades  and  the  number  of  fixations,  these  two  variables 
r^resented  the  same  phenomenon.  Table  K-9  presents  the  correlations  among  general  eye 
movement-related  variables.  What  is  strikmg  about  the  table  of  correlations  is  the  apparent 
independence  of  the  various  measures.  Given  the  integrated  nature  of  vision,  it  would  not  have 
been  surprising  to  see  more  redundancy  in  these  measures. 
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Table  K-9.  Correlations  Between  General  Eye  Movement  Related  Variables 


Saccade  Number 

Saccade  Mean  Duration 

Saccade  Mean  Distance 

Eye  Motion  Workload 

1 

'Z 

S 

Blink  Mean  Duration 

Blink  Mean  Distance 

Pupil  Number 

Pupil  Mean  Diameter 

Pupil  Motion  Workload 

1 

e 

o 

1 

Fixation  Mean  Duration 

Fixation  Area  Mean 

Visual  Efficiency 

Saccade  Number 

. 1 

-0.1 

-0.35 

0.06 

0.47 

-0.29 

-0.1 

0.77 

0,19 

-0.18 

0.99 

-0,17 

-0.03 

-0.13 

Saccade  Mean  Duration 

1 

0.6 

0.48 

-0.28 

0.19 

0.3 

-0.21 

•0.25 

0.51 

-0.12 

-0.4 

-0.16 

-0.74 

Saccade  Mean  Distance 

1 

0.76 

-0.2 

0.17 

0.3 

-0.42 

0.14 

0.15 

-0.35 

-0.34 

-0.26 

-0.49 

Eye  Motion  Workload 

1 

-0.08 

-0.16 

0.19 

-0.26 

0.2 

0 

0.02 

-0.57 

-0.38 

-0.66 

Blink  Number 

1 

-0.09 

-0.11 

0.37 

-0.11 

-0.11 

0.48 

-0.11 

0.13 

0.08 

Blink  Mean  Duration 

1 

0.07 

-0.35 

-0.15 

0.15 

1  -0.28 

-0.15 

-0.13 

-0.22 

Blink  Mean  Distance 

1 

-0.19 

0.14 

0.04 

1  -0.08 

-0.25 

-0.1 

-0.3 

Pi:pil  Number 

1 

0.2 

-0.18 

1  0.79 

0.38 

0.31 

0.34 

Piq>il  Mean  Diameter 

1 

-0.1 

0.18 

-0.06 

-0.04 

-0,13 

Pipil  Motion  Workload 

1 

-0.16 

-0.23 

-0.08 

-0.34 

Fixation  Number 

1 

-0.16 

-0.04 

-0.1 

Fixation  Mean  Duration 

1 

0.49 

0.81 

Fixation  Area  Mean 

1 

1  0.45 

\^sual  Efficiency 

1 

Scene  Plane  Fixations 


The  second  level  of  eye  movement  data  included  fixations,  broken  down  by  the  scene  plane  on 
which  they  rested.  Table  K-10  and  Table  K-1 1  present  the  scene  plane  scanning  variables  and 
the  distribution  of  total  fixation  times  across  scene  planes  respectively.  The  participants  fixated 
58%  of  the  total  time  or  75%  of  the  fixation  time  on  the  radarscope.  The  mean  duration  of  the 
radarscope  fixations  was  620  ms.  Participants  spent  only  0.5%  of  the  total  time  or  0.6%  of  the 
fixation  time  on  fixations  on  the  ATWTT  device.  The  mean  ATWIT  fixation  duration  was  610 
ms.  Fixations  on  the  flight  strip  bay  accoimted  for  2.2%  of  the  total  time  or  2.9%  of  the  fixation 
time.  The  mean  duration  of  flight  strip  bay  fixations  was  320  ms.  Participants  fixated  on  the 
keyboard/mouse  Area  for  17%  of  the  total  time  or  21.7%  of  the  fixation  time.  The  mean  duration 
of  the  keyboard/mouse  area  fixations  was  450  ms. 
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Table  K-10.  Scene  Plane  Visual  Scanning  Variables  (n=864) 


Variable  Label 

Mean 

SD 

Units 

Radarscope  Number 

290.96 

53.13 

Radarscope  Mean  Duration 

620.00 

110.00 

ms 

Radarscope  Total  Duration 

175.46 

26.62 

seconds 

ATWIT  Number 

2.84 

2.75 

ATWrr  Mean  Duration 

610.00 

410.00 

ms 

ATWIT  Total  Duration 

1.49 

1.00 

seconds 

Flight  Strips  Number 

19.03 

21.31 

Flight  Strips  Mean  Duration 

320.00 

130.00 

ms 

Flight  Strips  Total  Duration 

6.74 

9.20 

seconds 

Keyboard/Mouse  Number 

113.16 

42.70 

Keyboard/Mouse  Mean  Duration 

450.00 

98.00 

ms 

Keyboard/Mouse  Total  Duration 

51.00 

20.57 

seconds 

Note  that  the  amount  of  time  spent  looking  at  the  ATWIT  device  was  on  average  1.5  seconds  per 
5-minute  interval  (Table  K- 1 1 ). 


Table  K-1 1 .  Cumulative  Fixation  Duration  for  the  4  Scene  Planes;  Radarscope,  ATWTT  Panel, 

Flight  Strip  Bay,  and  Keyboard  Mouse  Area 


Variable  Label 

Mean 

SD 

Percent 
Total  Time 

Percent 
Fixation  Time 

Radarscope  Total  Duration 

175.46 

26.62 

58.48 

74.76 

ATWIT  Total  Duration 

1.49 

1.00 

0.50 

0.64 

Flight  Strips  Total  Duration 

6.74 

9.20 

2.25 

2.87 

Keyboard/Mouse  Total  Duration 

51.00 

20.57 

17.00 

21.73 

The  correlations  between  the  number  and  duration  of  fixations  on  different  scene  planes  were 
low.  The  highest  correlation  occurred  between  the  mean  duration  of  fixations  on  the  flight  strip 
bay  and  the  duration  of  fixations  on  the  keyboard  area  (r  =.35).  The  distribution  of  the  fixations 
across  the  scene  planes  therefore  did  not  seem  to  follow  a  fixed  pattern. 

Padarscope  Fixations 

The  third  and  most  detailed  level  of  analysis  focused  on  object  fixations  on  the  Plan  View 

Display.  The  main  information  display  in  air  traffic  control  is  the  PVD  or  radarscope.  The 

objects  of  fixations  on  the  PVD  were  data  blocks,  CA/LA,  other  statics  (airports,  fixes,  VORs, 
etc.),  preview  area,  system  area,  and  the  tablist  area.  The  researchers  calculate  the  mean 
duration,  number  of  fixations,  and  the  total  duration  for  each  of  these  categories.  The  researchers 
also  expressed  the  total  duration  in  percentage  of  the  total  time,  percentage  of  the  total  fixation 
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time,  and  percentage  of  the  total  fixation  time  on  the  radarscope.  Table  K-12  presents  the 
number  and  duration  of  the  fixations  on  radarscope  objects.  Table  K-12  presents  the  overall 
mean  fixation  durations  and  their  standard  deviations. 

The  participants  spent  on  average  92%  of  their  time  on  the  radarscope  looking  at  aircraft  data 
blocks.  On  average,  ATCSs  looked  at  the  aircraft  representations  on  the  radarscope  251  times  in 
a  5-minute  interval,  or  roughly  50  times-per-minute.  The  average  fixation  duration  on  aircraft 
representations  was  660  ms.  The  CA/LA  area  accoimted  for  a  negligible  small  percentage  of  the 
fixated  time  (visited  approximately  once  every  20  minutes).  ATCSs  looked  at  CA/LA  with 
average  fixation  duration  of  only  30  ms.  ATCSs  rested  their  gaze  on  static  objects  in  2%  of  the 
time  fixated  on  the  radarscope.  In  a  5-  minute  interval,  8  of  the  participants’  fixations  rested  on 
static  objects.  The  average  duration  of  these  fixations  was  150  ms.  Participants  fixated  on  the 
preview  area  in  2%  of  the  time  of  the  radarscope  fixations.  The  participants  looked  at  the 
preview  area  an  average  of  approximately  six  times  every  5  minutes.  The  mean  duration  of  the 
fixations  on  the  preview  was  on  average  150  ms.  ATCSs  fixated  on  the  system  area  2%  of  the 
time  of  the  radarscope  fixations  (visited  an  average  of  approximately  9  times).  The  fixations  on 
the  systems  area  lasted  380  ms.  Lastly,  ATCSs  fixated  on  the  tab  list  in  a  negligible  small 
percentage  of  the  time  of  the  radarscope  fixations  (visited  on  average  of  three  times  per  5 
minutes).  The  fixations  on  the  tab  list  lasted  160  seconds. 


Table  K-12.  Radarscope  Objects 


Variable  Label 

Mean 

SD 

Units 

Data  Block  Mean  Duration 

660.00 

130.00 

Ms 

Data  Block  Number 

250.72 

59.56 

Data  Block  Total  Duration 

161.07 

33.99 

Seconds 

CA/LA  Mean  Duration 

30.00 

110.00 

Ms 

CA/LA  Number 

0.23 

0.81 

Frequency 

CA/LA  Total  Duration 

0.07 

0.27 

Seconds 

Other  Statics  Mean  Duration 

150.00 

170.00 

Ms 

Other  Statics  Nmnber 

8.34 

17.24 

Other  Statics  Total  Duration 

2.86 

6.71 

Seconds 

Preview  Mean  Duration 

400.00 

320.00 

Ms 

Preview  Number 

5.67 

7.22 

Preview  Total  Duration 

2.69 

3.95 

Seconds 

System  Mean  Duration 

380.00 

220.00 

Ms 

System  Number 

8.78 

8.89 

System  Total  Duration 

3.92 

4.95 

Seconds 

Tab  list  Mean  Duration 

160.00 

280.00 

Ms 

Tab  list  Number 

1.54 

3.16 

Tab  list  Total  Duration 

0.64 

1.56 

Seconds 
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Table  K-13  clearly  shows  that  the  most  important  elements  on  the  radarscope  were  aircraft.  On 
average,  aircraft  fixations  constituted  92%  of  the  fixation  time  on  the  radarscope. 


Table  K-13.  Cumulative  Fixation  Duration  on  Objects  on  the  Radarscope 


Variable  Label 

N 

Mean 

SD 

Percent 
Total  Time 

Percent  Radarscope 
Fixation  Time 

Aircraft  Total  Duration 

864 

161.07 

33.99 

54.00 

69.00 

92.00 

CA/LA  Total  Duration 

864 

0.07 

0.27 

0.00 

0.00 

0.00 

Other  Statics  Total  Duration 

864 

2.86 

6.71 

1.00 

1.00 

2.00 

Preview  Total  Duration 

864 

2.69 

3.95 

1.00 

1.00 

2.00 

System  Total  Duration 

864 

3.92 

4.95 

1.00 

2.00 

2.00 

Tab  list  Total  Duration 

864 

0.64 

1.56 

0.00 

0.00 

0.00 

ATwrr 


Equipment  failure  caused  the  loss  of  data  for  one  simulation  scenario.  For  this  simulation,  the 
researchers  substituted  the  overall  mean  value  for  the  ATWIT  variables  for  each  interval.  The 
data  set  used  for  the  descriptive  statistics  contained  96  observations.  The  ATWTT  ratings  showed 
a  trend  as  a  fimction  of  interval  number  (Table  K-14),  reflecting  the  buildup  of  traffic  during  the 
first  10-15  minutes  of  the  scenarios. 
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Table  K-14.  ATWIT  Ratings  as  a  Function  of  Simulation  Interval  (n=96) 


Interval 

Low  Load 

Means  | 

SDs 

Means  | 

SDs 

Means 

SDs 

1 

1.50 

1.84 

1.67 

0.70 

1.58 

1.38 

2.00 

2.57 

3.67 

2.35 

2.83 

2.58 

2.25 

2.03 

4.83 

2.57 

3.54 

2.63 

M 

3.13 

2.49 

4.67 

1.97 

3.90 

2.35 

3.38 

2.58 

4.38 

2.04 

3.88 

2.36 

z 

3.92 

2.12 

4.79 

2.13 

4.35 

2.15 

3.75 

2.31 

4.92 

2.24 

4.33 

2.33 

8 

1.67 

1.90 

5.21 

2.52 

3.44 

2.84 

9 

2.17 

3.03 

5.50 

2.87 

3.83 

3.37 

1 

1.13 

0.34 

1.42 

1.84 

1.27 

1.32 

1.54 

0.72 

2.63 

2.46 

2.08 

1.88 

2.08 

1.89 

3.79 

2.32 

2.94 

2.26 

2.50 

1.96 

5.50 

2.13 

4.00 

2.53 

1 

2.33 

1.31 

6.67 

1.95 

4.50 

2.74 

2.67 

1.99 

6.71 

2.05 

4.69 

2.86 

2.38 

2.04 

6,33 

2.43 

4.35 

2.99 

8 

2.46 

1.98 

5.38 

2.34 

3.92 

2.60 

9 

3.13 

2.92 

3.67 

3.31 

3.40 

3.10 

1 

1.31 

1.32 

1.54 

1.38 

1.43 

1.35 

1.77 

1.88 

3.15 

2.44 

2.46 

2.28 

2.17 

1.94 

4.31 

2.48 

3.24 

2.46 

2.81 

2.24 

5.08 

2.07 

3.95 

2.43 

2.85 

2.09 

5.52 

2.29 

4.19 

2.56 

3.29 

1 

2.13 

5.75 

2.28 

4.52 

2.52 

3.06 

2.26 

5.63 

2.42 

4.34 

2.66 

8 

2.06 

1.96 

5.29 

2.41 

3.68 

2.72 

9 

2.65 

2.99 

4.58 

3.20 

3.61 

3.23 
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The  ATWrr  latencies  showed  a  similar  increase  at  the  onset  of  a  scenario,  although  the  effect 
was  not  as  pronounced  as  in  the  ATWIT  ratings  themselves  (Table  K-15). 

Table  K-15.  ATWIT  Latencies  as  a  Fimction  of  Simulation  Interval  (n=96) 


Interval 

Low  Load 

Means 

SDs 

Means  | 

SDs 

Means 

SDs 

1 

3.13 

3.99 

2.75 

1.57 

2.94 

3.01 

2 

4.75 

5.30 

3.79 

5.30 

4.27 

5.27 

3 

3.33 

4.21 

5.13 

6.47 

4.23 

5.47 

4 

6.13 

5.97 

4.42 

4.78 

5.27 

5.42 

2 

5 

5.00 

6.07 

2.88 

4.18 

3.94 

5.27 

6 

3.83 

4.40 

3.38 

4.27 

3.60 

4.30 

7 

4.79 

5.73 

2.83 

3.71 

3.81 

4.88 

8 

4.29 

5.24 

4.46 

5.08 

4.38 

5.11 

9 

5.50 

6.84 

6.29 

7.09 

5.90 

6.90 

1 

2.54 

1.53 

2.96 

3.94 

2.75 

2.96 

2 

2.67 

2.35 

4.29 

5.47 

3.48 

4.25 

3 

4.54 

4.42 

5.42 

6.37 

4.98 

5.44 

4 

4.88 

4.55 

4.54 

6.20 

4.71 

5.38 

o 

5 

4.96 

5.09 

4.21 

5.21 

4.58 

5.11 

z 

6 

4.46 

4.94 

4.88 

5.71 

4.67 

5.29 

7 

4.13 

4.32 

3.50 

5.34 

3.81 

4.81 

8 

4.92 

4.76 

3.13 

4.18 

4.02 

4.53 

9 

5.42 

6.32 

6.33 

6.98 

5.88 

6.60 

1 

2.83 

3.01 

2.85 

2.97 

2.84 

5.17 

2 

3.71 

4.19 

4.04 

5.34 

2.97 

4.14 

3 

3.94 

4.31 

5.27 

6.35 

3.88 

4.82 

4 

5.50 

5.29 

4.48 

5.48 

4.78 

3.81 

5 

4.98 

5.54 

3.54 

4.72 

4.60 

4.82 

6 

4.15  . 

4.64 

4.13 

5.05 

5.44 

4.20 

7 

4.46 

5.03 

3.17 

4.56 

4.99 

4.80 

8 

4.60 

4.96 

3.79 

4.65 

5.38 

5.89 

9 

5.46 

6.51 

6.31 

6.96 

4.26 

6.72 

Performance  Measures 

The  data  reduction  and  analysis  (DRA)  program  reduced  the  simulator  data  files,  the  simulation 
pilot  command  files,  the  push-to-talk  (communication),  and  the  ATWIT  files  to  a  set  of  41 
variables.  These  variables  were  divided  into  Conflict,  Complexity,  Error,  Communications,  and 
Task  Load  variables . 

Conflicts 

The  DRA  module  calculated  the  number  and  duration  of  standard,  terminal,  longjtudin^,  and 
parallel  conflicts.  The  DRA  module  originally  reduced  data  of  experimaits  with  IFR  a^raft  ILS 
approaches  only.  In  the  current  experiment,  however,  both  IFR  and  VFR  aircraft  and  visual 
approaches  were  present.  These  variables  are  indicators  of  how  close  a  ATCS  works  traffic. 
They  are  not  the  number  of  times  an  ATCS  violated  separation  requirements. 
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Table  K-16  summarizes  the  means  and  standard  deviations  for  the  conflict  related  variables.  On 
average,  there  were  3.2  standard  conflicts  per  45-minute  simulation.  These  conflicts  lasted 
approximately  4  minutes  and  20  seconds.  Terminal  Conflicts  occurred  1.6  times  per  scenario  for 
a  total  of  about  1  minute  and  20  seconds.  Less  than  one  longitudinal  conflict  occurred  per 
scenario,  on  average  lasting  less  than  50  seconds.  No  Parallel  Conflict  Information  was 
recorded.  The  number  of  recorded  Between  Sector  Conflicts  was  approximately  5.5  per  45- 
minute  scenario.  The  cumulative  Duration  of  Between  Sector  Conflicts  within  a  scenario  was 
approximately  5  minutes  and  45  seconds.  The  Closest  Point  of  Approach  was  less  than  2800  feet 
with  a  Horizontal  Separation  of  less  than  2200  feet  and  a  vertical  separation  of  less  than  480  feet. 
The  aircraft  Proximity  Index  during  this  experiment  averaged  almost  27.5. 


Table  K-16.  Mean  and  SDs  of  DRA  Variables  Related  to  Task  Load  as  a  Function  of  Task  Load 

and  Visual  Noise 


Low  Load 

Means 

SDs 

Means 

SDs 

Means 

SDs 

No  Standard  Conflicts 

2.21 

1.44 

0.83 

1.05 

2.76 

1.77 

Dur  Standard  Conflicts 

222.42 

629.98 

150.33 

482.71 

449.88 

No  Terminal  Conflicts  ( 3/500) 

0.17 

0.38 

Dur  Terminal  Conflicts  (3/500) 

8.17 

15.35 

3.38 

10.24 

19.40 

41.23 

ca 

No  Longitudinal  Conflicts 

0.33 

0.48 

1 

Dur  Longitudinal  Conflicts 

36.83 

69.92 

17.75 

27.06 

44.42 

68.58 

o 

No  Between  Sector  Conflicts 

2.63 

2.24 

3.33 

1.17 

4.21 

Dur  Between  Sector  Conflicts 

141.04 

169.67 

130.98 

282.28 

225.95 

Closest  Point  of  Approach  (Feet) 

4393.92 

5804.33 

3728.83 

5299.35 

2595.69 

4473.78 

CPA  Horizontal  Separation 

4223.00 

5900.05 

3563.75 

5385.99 

2358.60 

4567.23 

CPA  Vertical  Separation 

556.42 

493.17 

275.18 

516.99 

229.31 

Aircraft-  Proximity-  Index  (0-100) 

19.54 

26.67 

28.08 

29.32 

No  Standard  Conflicts 

3.32 

1.92 

6.79 

3.54 

3.81 

3.97 

Dur  Standard  Conflicts 

139.46 

114.57 

523.67 

436.68 

337.00 

492.87 

No  Terminal  Conflicts  (  3/500) 

5.25 

2.29 

2.71 

3.04 

Dur  Terminal  Conflicts  (3/500) 

54.53 

306.63 

190.18 

155.00 

203.05 

No  Longitudinal  Conflicts 

1.12 

1.50 

1.18 

0.92 

1.07 

o 

CO 

Dur  Longitudinal  Conflicts 

67.84 

77.71 

72.60 

47.73 

62.09 

:! 

No  Between  Sector  Conflicts 

5.79 

10.21 

2.70 

6.77 

4.04 

Dur  Between  Sector  Conflicts 

417.56 

648.67 

181.86 

409.17 

288.38 

Closest  Point  of  Approach  (Feet) 

191  AS 

678.83 

599.63 

318.23 

2164.23 

4036.38 

CPA  Horizontal  Separation 

494.20 

749.25 

430.79 

276.28 

1997.27 

4091.36 

CPA  Vertical  Separation 

477.56 

395.83 

207.43 

444.50 

246.03 

Aircraft-  Proximity-  Index  (0- 1 00) 

wem 

37.75 

32.41 

32.92 

30.96 

No  Standard  Conflicts 

1.52 

1.43 

5.05 

3.32 

3.29 

3.10 

Dur  Standard  Conflicts 

186.38 

556.39 

331.56 

370.71 

258.97 

475.89 

No  Terminal  Conflicts  (  3/500) 

0.23 

0.42 

2.96 

2.87 

1.60 

2.46 

Dur  Terminal  Conflicts  (3/500) 

5.77 

13.13 

168.63 

196.47 

87.20 

160.88 

No  Longitudinal  Conflicts 

0.40 

0.64 

1.26 

1.16 

0.83' 

1.03 

Dur  Longitudinal  Conflicts 

27.29 

53.33 

64.86 

70.71 

46.07 

65.09 

No  Between  Sector  Conflicts 

2.98 

1.80 

8.00 

3.25 

5.49 

3.64 

Dur  Between  Sector  Conflicts 

158.33 

135.13 

533.11 

229.02 

345.72 

265.46 

Closest  Point  of  Approach  (Feet) 

4061.38 

5508.40 

698.54 

533.90 

2379.96 

4243.76 
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Complexity 


The  average  system  activity  was  16.  The  average  number  of  altitude  changes  was  approximately 
25  per  scenario.  A  45-minute  simulation  contained  approximately  28  heading  changes.  ATCSs 
instructed  aircraft  to  change  their  speeds  less  than  two  times  per  scenario.  Table  K-17 
summarizes  the  means  and  standard  deviations  for  complexity  related  variables. 

Table  K-17.  Means  and  SDs  of  Complexity  Related  Variables  as  a  Function  of  Task  Load  and 

Visual  Noise 


Low  Load 
Means  |  SDs 

High  Load 
Means  |  SDs 

Means 

SDs 

Average  System  Activity 

4.56 

0.90 

13.19 

2.68 

7.47 

4.10 

'o 

Number  of  Altitude  Changes 

19.42 

5.46 

17.79 

5.58 

25.32 

10.01 

o 

Number  of  Heading  Changes 

19.88 

10.12 

18.71 

6.57 

28.39 

14.50 

z 

Number  of  Speed  Changes 

2.17 

1.97 

0.29 

0.55 

1.80 

1.72 

Average  System  Activity 

10.37 

3.99 

36.15 

6.94 

24.67 

12.71 

Number  of  Altitude  Changes 

31.22 

10.11 

30.29 

11.07 

24.04 

10.73 

1 

Number  of  Heading  Changes 

36.90 

13.27 

43.71 

15.39 

31.21 

17.22 

Number  of  Speed  Changes 

1.43 

1.37 

2.92 

3.05 

1.60 

2.54 

Average  System  Activity 

8.88 

4.79 

23.26 

14.18 

16.07 

12.77 

Number  of  Altitude  Changes 

18.60 

5.52 

30.76 

10.50 

24.68 

10.34 

Number  of  Heading  Changes 

19.29 

8.46 

40.30 

14.63 

29.80 

15.90 

Number  of  Speed  Changes 

1.23 

1.72 

2.18 

2.46 

1.70 

2.16 

Error 

The  error-related  variables  contributed  relatively  little  to  insight  in  the  performance  of  the 
ATCSs  in  this  study.  The  simulation  pilots  did  not  execute  missed  approaches.  The  ATCSs  nor 
did  not  issue  hand-offs  outside  the  sector  boundary.  The  number  and  duration  of  turns  and  holds 
were  extremely  low.  Interestingly  enough,  the  DRA  found  an  average  of  five  Start  Point  Delaj^ 
with  an  average  Start  Point  Delay  Duration  of  35  seconds.  Most  likely,  this  was  due  to  delays  in 
the  simulation  software  because  the  current  study  did  not  contain  aircraft  that  needed  a  manual 
release.  ATCSs  did  not  have  the  option  to  hold  traffic  at  the  airport  as  a  tool  to  control  traffic 
flow.  Table  K-18  summarizes  the  means  and  standard  deviations  for  error  related  variables. 


Table  K-18.  Mean  and  Standard  Deviation  of  Error  Related  Variables 


Variable 

Mean 

SD 

No  Missed  Approaches 

0.00 

0.00 

No  Hand-offe  Outside  Boundary 

0.00 

0.00 

No  Tum/Hold  Delays 

0.10 

0.30 

Dur  Tum/Hold  Delays 

5.59 

23.05 

No  Start  Point  Delays 

5.00 

2.27 

Dur  Start  Point  Delay 

35.67 

208.80 
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Communications 


The  average  number  of  ATCS  messages  per  simulation  run  was  approximately  36,  and  die 
number  of  pilot  keystrokes  totaled  about  480.  Table  K-19  summarizes  the  means  and  standard 
deviations  for  communications-related  variables. 


Table  K-19.  Mean  and  Standard  Deviation  of  Communications-Related  Variables 


Variable 

Mean 

SD 

No  Ground-to-Air  Contacts 

4.11 

28.05 

Dur  Ground-to-Air  Contacts 

18.07 

115.00 

No  ATCS  Messages 

35.76 

17.07 

No  Pilot  Message  Key  Strokes 

479.36 

213.29 

Task  Load 

The  average  number  of  aircraft  handled  was  approximately  26.  The  cumulative  time  ATCSs  had 
aircraft  under  control  averaged  almost  19,800  seconds  or  5  hours  and  30  minutes.  The  aircraft 
xmder  control  flew  an  average  of  a  cumulative  distance  of  1600  miles.  On  average,  the  number 
of  arrivals,  departures,  and  accepted  hand-offs  were  5.5, 7.5,  and  10.5,  respectively.  Aircraft 
arrived  every  2  minutes  and  40  seconds  and  departed  every  6  minutes  and  30  seconds.  Table  K- 
20  summarizes  the  means  and  standard  deviations  for  task  load-related  variables. 


Table  K-20.  Mean  and  Standard  Deviation  of  Task  Load-Related  Variables 


Variable 

Mean 

SD 

No  Aircraft  Handled 

26.36 

8.01 

Time  Under  Control 

19734.51 

7330.47 

Distance  Flown 

1624.06 

3404.61 

No  Completed  Flights 

7.61 

2.00 

No  Arrivals 

5.56 

3.07 

Ave  Arrival  Interval  (Seconds) 

221.32 

133.54 

No  Departures 

7.36 

2.63 

Ave  Departure  Interval  (Seconds) 

390.34 

155.98 

No  Hand-offs  Accepted 

10.46 

3.77 

Hand-off  Accept  Delay  Time 

0.00 

0.00 
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Appendix  L 

Detailed  Results  of  Selected  Statistical  Analyses 


Table  L-1.  MANOVA  Results  for  General  Post-Scenario  Questions 


Effect 

Wilk’s'  Lambda 

F 

NumDF 

DenDF 

/><.05 

Load  X  Visual  noise 

.413 

2.663 

8 

15 

.0486 

Load 

.021 

89.261 

8 

15 

.000 

No  Visual  noise 

.041 

44.301 

8 

15 

.000 

Visual  noise 

.078 

22.082 

8 

15 

.000 

Visual  noise 

.370 

3.191 

8 

15 

.025 

Low  Load 

.531 

1.658 

8 

15 

.190 

High  Load 

.414 

2.659 

8 

15 

.049 

Table  L-2.  Load  and  Visual  Noise  Interaction  on  General  Post-Scenario  Questions 


Ques# 

Variable 

DF 

Type  in  SS 

Mean  Square 

F 

p<.05 

1 

Realism 

1 

1.168 

1.168 

0.650 

.429 

2 

Representativeness 

1 

4.175 

4.175 

2.150 

.157 

3 

ATWTT  Interference 

1 

0.200 

0.200 

0.590 

.452 

4 

Oculometer  Interference 

1 

0.830 

0.830 

0.790 

.383 

5 

Simulation  Pilot  Performance 

1 

2.625 

2.625 

3.110 

.092 

7 

Working  Hard? 

1 

12.676 

12.676 

9.240 

.006 

8 

Control  Quality 

1 

7.353 

7.353 

8.190 

.009 

13 

Difficulty 

1 

11.908 

11.908 

11.210 

.003 

Table  L-3.  Effect  of  Task  Load  on  Individual  General  Post-Scenario  Questions 


Ques# 

Variable 

DF 

Type  mss 

Mean  Square 

F 

p<.05 

1 

Realism 

1 

5.709 

5.709 

2310 

.138 

2 

Representativeness 

1 

28.144 

28.144 

8.170 

.009 

3 

ATWIT  Interference 

1 

3.234 

3.234 

9.900 

.005 

4 

Oculometer  Interference 

1 

0.858 

0.858 

0.530 

Ali 

5 

Simulation  Pilot  Performance 

1 

5.100 

5.100 

6.590 

.018 

7 

Working  Hard? 

1 

349.285 

349.285 

296.660 

.000 

8 

Control  Quality 

1 

29.739 

29.739 

14.440 

.001 

13 

Difficulty  1 

1 

400.941 

400.941 

263.880 

.000 

Table  L-4.  MANOVA  Results  for  Post-Scenario  SA  Related  Questions 


Effect 

Wilk’s' Lambda 

F 

NumDF 

DenDF 

;7<.05 

Visual  noise 

.552 

3.863 

4 

19 

.018 

Load 

.316 

4 

19 

.000 

Load  X  Visual  noise 

.668 

2.366 

4 

19 
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Table  L-5.  Effect  of  Task  Load  on  Individual  SA  Related  Post-Scenario  Questions 


Variable 

DF 

Type  in  SS 

Mean  Square 

F 

A 

O 

Overall  SA 

1 

37.816 

37.816 

25.190 

.000 

Current  ACFT  Location  SA 

1 

48.525 

48.525 

42.980 

.000 

Projected  ACFT  Location  SA 

1 

41.690 

41.690 

32.850 

.000 

Potential  Violations  SA 

1 

22.224 

22.224 

13.030 

.002 

Table  L-6.  Effect  of  Visual  noise  on  Individual  SA  Related  Post-Scenario  Questions 


Variable 

DF 

Type  m  SS 

Mean  Square 

F 

p<.05 

Overall  SA 

1 

4.062 

4.062 

3.950 

.059 

Current  ACFT  Location  SA 

1 

6.905 

6.905 

5.460 

.029 

Projected  ACFT  Location  SA 

1 

6.374 

6.374 

5.830 

.025 

Potential  Violations  SA 

1 

13.358 

13.358 

14.630 

.001 

Table  L-7.  MANOVA  Results  of  Post-Scenario  TLX  Items 


Effect 

Wilk’s'  Lambda 

F 

NumDF 

DenDF 

p  <  .05 

Load  X  Visual  noise 

.545 

2.363 

6 

17 

.076 

Load 

.060 

45.175 

6 

17 

.000 

Visual  noise 

.518 

2.633 

6 

17 

.054 

Table  L-8.  Effect  of  Task  Load  on  Individual  TLX  Items  in  the  Post-Scenario  Questionnaire 


Variable 

Type  mss 

Mean  Square 

F 

p<.05 

Mental  Demand 

1 

328.716 

328.716 

222.270 

.000 

Physical  Demand 

m 

150.211 

150.211 

41.910 

.000 

Tenq)oral  Demand 

1 

242.671 

242.671 

99.950 

.000 

Performance 

1 

15.394 

15.394 

8.720 

.007 

Effort 

1 

44.425 

44.425 

23.840 

.000 

Frustration 

170.274 

170.274 

80.050 

.000 

Table  L-9.  MANOVA  Results  for  Mean  and  Maximum  ATWIT  Ratings 


Effect 

Wilk’s' Lambda 

F 

NumDF 

DenDF 

p<.05 

Load  X  Visual  noise 

.702 

4.453 

2 

21 

.024 

Load 

.159 

55.738 

2 

21 

.000 

No  Visual  noise 

.093 

102.960 

2 

21 

.000 

Visual  noise 

.330 

21.304 

2 

21 

.000 

Visual  noise 

.988 

0.129 

2 

21 

.879 

Low  Load 

.849 

1.861 

2 

21 

.180 

High  Load 

.856 

1.767 

2 

21 

.195 

Table  L-10.  Univariate  Interaction  of  Load  and  Visual  Noise  on  ATWIT  Mean  and  Maximum 


Load  X  Visual  noise 

DF 

Type  ffl  SS 

Mean  Square 

F 

p<.05 

ATWIT  Mean 

1 

1.960 

1.960 

3.690 

.068 

ATWrr  Maximum 

1 

29.739 

29.739 

9.190 

.006 
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Table  L-1 1 .  Univariate  Effect  of  Task  Load  on  ATWIT  Mean  and  Maximum 


Load 

Type  ni  SS 

Mean  Square 

F 

p  <  .05 

ATWIT  Mean 

1 

110.405 

110.405 

92.370 

.000 

ATWIT  Maximum 

1 

136.728 

136.728 

18.520 

.000 

Table  L-12.  MANOVA  Results  on  General  Eye  Movement  Characteristics 


Effect 

WiUc’s*  Lambda 

NumDF 

DenDF 

;7<.05 

Load  X  Visual  noise 

.617 

5 

18 

.095 

Visual  noise 

.900 

0.409 

5 

18 

.836 

Load 

.350 

6.680 

5 

18 

.001 

Table  L-13.  Effect  of  Task  Load  on  General  Eye  Movement  Characteristics 


Variable 

DF 

Type  in  SS 

Mean  Square 

F 

p  <  .05 

Number  of  Fixations 

1 

186469.740 

186469.740 

4.240 

.051 

Mean  Fixation  Duration 

'i 

0.004 

0.004 

2.170 

.155 

Mean  Fixation  Area 

1 

0.059 

0.059 

19.540 

.000 

Visual  Efficiency 

1 

0.001 

0.001 

0.890 

.357 

Mean  Saccade  Duration 

1 

0.000 

0.000 

0.160 

.692 

Mean  Saccade  Distance 

1 

0.407 

0.407 

4.310 

.050 

Eye  Motion  Workload 

1 

0.217 

0.217 

0.590 

.451 

Number  of  Blinks 

1 

119114.555 

119114.555" 

3.040 

.095 

Mean  Blink  Duration 

1 

0.000 

0.000 

0.000 

.960 

Mean  Blink  Distance 

1 

4.224 

4.224 

0.610 

.442 

Mean  Pupil  Diameter 

1 

0.215 

0.215 

0.920 

.347 

Pupil  Motion  Workload 

1 

0.000 

0.000 

0.440 

.516 

Table  L-1 4.  MANOVA  Results  on  Scene  Plane  Fixation  Characteristics 


Effect 

Wilk’s'  Lambda 

F 

NumDF 

DenDF 

p  <  .05 

Load  X  Visual  noise 

.251 

14.200 

4 

19 

.000 

Visual  noise 

.639 

2.700 

4 

19 

.063 

Low  Load 

.460 

5.580 

4 

.004 

High  Load 

.415 

5.070 

5 

18^ 

.005 

Load 

.110 

38.490 

4 

19 

.000 

No  Visual  noise 

.213 

17.595 

4 

19 

.000 

Visual  noise 

.119 

26.596 

5 

18 

.000 
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Table  L-15.  Effect  of  Task  Load  and  Visual  Noise  Interaction  on  Scene  Plane  Fixations 


Variable 

DF 

Type  III  SS 

Mean  Square 

F 

p<.05 

Radarscope  Fixations 

1 

248482.300 

248482.300 

15.620 

.001 

Radarscope  Mean  Duration 

1 

0.026 

0.026 

.000 

Flight  Strip  Bay  Fixations 

1 

25091.170 

25091.170 

14.720 

.001 

Flight  Strip  Bay  Mean  Duration 

1 

0.001 

0.001 

0.460 

.504 

ATWrr  Fixations 

1 

15.583 

15.583 

0.110 

.742 

ATWIT  Mean  Duration 

1 

0.008 

0.008 

0.450 

.508 

Keyboard  Area  Fixations 

1 

19481.580 

19481.580 

1.060 

.316 

Keyboard  Area  Mean  Duration 

1 

0.021 

0.021 

8.520 

.008 

Table  L-16.  MANOVA  Results  on  Radarscope  Fixations 


Effect 

Wilk’s'  Lambda 

F 

NumDF 

DenDF 

p<.05 

Load  X  Visual  noise 

.151 

19.198 

5 

17 

.000 

Visual  noise 

.157 

14.320 

6 

16 

.000 

Load 

.151 

15.034 

6 

16 

.000 

Table  L-17.  biteraction  Effects  of  Task  Load  and  Visual  Noise  for  Radar  Object  Related 

Fixations 


Variables 


System  Area  Fixations 


System  Area  Mean  Duration 


Static  Object  Fixations 


/?<  .05 

5273.824 

10.540 

.004 

0.017 

2.920 

.102 

Static  Object  Mean  Duration 

0.059 

12.910 

.002 

Tab  List  Fixations 

IBI 

1633.818 

1633.818 

20.850 

.000 

Tab  List  Mean  Duration 

1 

0.117 

0.117 

6.470 

.019 

Preview  Area  Fixations 

1 

1997.909 

1997.909 

4.100 

.055 

Preview  Area  Mean  Duration 

1 

0.008 

0.008 

1.000 

.329 

Aircraft  Fixations 

1 

948841.000 

948841.000 

46.850 

.000 

Aircraft  Mean  Duration 

1 

0.059 

0.059 

28.220 

.000 

Table  L- 1 8.  Effects  of  Task  Load  for  Radar  Obj ect  Related  Fixations 


Variables 

lES 

Type  in  SS 

Mean  Square 

F 

;j<.05 

System  Area  Fixations 

IB 

12947.480 

.  12947.480 

22.380 

.000 

System  Area  Mean  Duration 

1 

0.188 

0.188 

44.090 

.000 

Static  Object  Fixations 

1 

69790.560 

69790.560 

47.500 

IIIIQ 

Static  Object  Mean  Duration 

1 

0.005 

0.005 

0.600 

.448 

Tab  List  Fixations 

B 

30.168 

30.168 

0.780 

.386 

Tab  List  Mean  Duration 

0.067 

1.740 

.201 

Preview  Area  Fixations 

1 

10293.770 

10293.770 

13.700 

.001 

Preview  Area  Mean  Duration 

1 

0.001 

0.001 

0.110 

.742 

Aircraft  Fixations 

1 

621127.500 

621127.500 

11.760 

.002 

Aircraft  Mean  Duration 

1 

0.006 

0.006 

1.790 

.194 

Table  L-19.  Repeated  Measures  Anal}«is  Results  VFR  X  Condition  Interaction  for  General  Eye 

Movement  Characteristics 


Variables 

Wilk’s'  Lambda 

F 

NumDF 

DenDF 

p<  .05 

Number  of  Fixations 

.833 

0.400 

4 

8 

.804 

Average  Fixation  Duration 

.541 

1.698 

4 

8 

.243 

Average  Fixation  Area 

.713 

0.806 

4 

8 

.555 

Visual  Efficiency 

.491 

2.076 

4 

8 

.176 

Average  Saccade  Duration 

.305 

4.558 

4 

8 

.033 

Average  Saccade  Distance 

.438 

2.567 

4 

8 

.119 

Eye  Motion  Workload 

.582 

1.436 

4 

8 

.307 

Number  of  Blinks 

.649 

1.082 

4 

8 

.426 

Average  Blink  Duration 

.460 

2.351 

4 

8 

.141 

Average  Blink  Distance 

.900 

0.224 

4 

8 

.918 

Pupil  Diameter 

.463 

2.319 

4 

8 

.145 

Pupil  Motion  Workload 

.463 

2.319 

4 

8 

.145 

Table  L-20.  Main  Effect  of  the  Presence  of  VFR  Intrusions  on  Eye  Movement  Variables 


Effect:  VFR 

Wilk’s'  Lambda 

F 

NumDF 

DenDF 

p  <  .05 

Number  of  Fixations 

.845 

2.018 

1 

11 

.183 

Average  Fixation  Duration 

.984 

0.179 

1 

li 

.681 

Average  Fixation  Area 

.898 

1.247 

1 

11 

.288 

Visual  Efficiency 

1.000 

0.000 

1 

11 

1.000 

Average  Saccade  Duration 

.939 

0.714 

1 

11 

.416 

Average  Saccade  Distance 

.976 

0.271 

1 

11 

.613 

Eye  Motion  Workload 

.952 

0.552 

1 

11 

.473 

Number  of  Blinks 

.844 

2.031 

1 

11 

.182 

Average  Blink  Duration 

.920 

0.951 

1 

11 

.350 

Average  Blink  Distance 

.976 

0.273 

1 

11 

.612 

Pupil  Diameter 

.850 

1.935 

1 

ii 

.192 

Pupil  Motion  Workload 

.731 

4.053 

1 

11 

.069 

Table  L-21.  Effect  of  Radarscope  Objects  on  Fixation  Duration 


Effect 

Wilk’s'  Lambda 

F 

NumDF 

DenDF 

o 

V 

Objects 

.018 

239.810 

4 

18 

.000 

Load 

.717 

8.300 

1 

21 

.009 

Visual  noise 

1.000 

0.000 

1 

21 

1.000 

Objects  X  Load 

.356 

8.158 

4 

18 

.001 

Objects  X  Visual  noise 

.894 

0.534 

4 

18 

.713 

Load  X  Visual  noise 

.392 

32.521 

1 

21 

.000 

Objects  X  Load  x  Visual  noise 

.557 

3.573 

4 

18 

.026 
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Appendix  M 

Other  Analyses  Opportunities 


The  combined  oculometry  and  simulator  data  sets  lend  themselves  to  other  forms  of  analyses. 

For  example,  for  each  fixation  this  data  point  included  the  targets  that  were  wilhin  a  circle  with  a 
2  inch  radius.  Researchers  can  calculate  a  transition  probability  (or  Markov)  matrix,  when 
choosing  targets  closest  to  the  fixation.  This  matrix  represents  the  probability  that  target  B 
follows  target  A  and  vice  versa.  SA  studies  have  suggested  that  ATCSs  group  the  aircraft  in 
their  airspace.  If  the  visual  scan  reflects  this  grouping,  the  Markov  matrix  will  reflect  this.  Ellis 
(1986)  suggested  that  experts  are  likely  to  scan  a  display  in  a  stratified  random  manner,  resulting 
in  a  symmetrical  Markov  matrix.  It  is  interesting,  that  these  analytical  techniques  were 
developed  for  stationary  objects.  In  our  facility,  the  objects  are  moving  targets  linking  the 
fixations  with  the  objects. 

Data  on  each  fixation  also  contains  information  on  its  coordinates.  Researchers  can  calculate  the 
number  of  fixations  per  segment  and  the  number  of  transitions  between  segments  by  breaking  up 
the  radarscope  into  polar  coordinates.  TRACON  ATCSs  will  indicate  that  they  scan  inside  out, 
that  is  ftom  the  center  of  the  radarscope  (the  aiiport)  to  the  edge  of  the  radarscope.  They  explain 
this  by  pointing  at  the  fact  that  the  airport  is  the  sink  of  the  problem,  all  arriving  aircraft  will 
converge  to  that  point  and  all  departing  aircraft  will  appear  at  that  point.  By  starting  to  solve 
problems  in  the  center  of  the  scope,  the  ATCS  starts  at  most  likely  the  hi^ly  congested  point. 
Using  a  Markov  matrix  based  on  polar  coordinate  segments,  researchers  visualize  the 
probabilities  of  moving  from  one  segment  to  the  next.  If  inside  out  scanning  exists,  this  will 
result  in  increased  probabilities  for  transitions  from  segments  that  are  closer  to  the  airport  or 
center  of  the  radarscope  than  for  segments  that  are  more  distant. 

Others  (Credeur  et  al.,  1993;  Hilbum  &  Parasuraman,  1996^)  have  used  a  division  of  the 
radarscope  in  sections  and  looked  at  transitions  between  these  sections.  The  division  of  the 
radarscope  in  sections  is  arbitrary.  Hilbiun  and  Parasuraman  used  a  grid  consisting  of  squares  to 
calculated  the  entropy  in  the  visual  scene  of  ATCSs  and  found  a  structured  scan.  By  basing  his 
divisions  purely  on  radarscope  location,  this  result  should  not  be  a  surprise.  After  all,  the 
airspace  structure  includes  airways  and  approach  patterns.  It  will  therefore  be  less  likely  that 
fixation  transitions  occur  between  areas  where  no  structural  elements  exist  and  areas  that  contain 
structural  elements.  A  study  by  Credeur  et  al.  (1993)  provides  a  better  approach.  This  study 
used  transitions  between  structural  elements. 

The  division  of  the  airspace  in  sections  in  reality  assigns  . fixations  to  bins  based  on  the  location 
on  the  radarscope.  There  are  alternatives  that  do  not  use  the  fixation  location.  The  alternative 
metiiods  may  shed  more  light  on  cognitive  processes  used  by  ATCSs  during  visual  sampling  of 


’  Dynamic  Decision  aiding  in  air  traffic  control:  A  bio-behavioral  analysis.  B.  Hilbum  and 
R.  Parasuraman,  1996,  Vivek,  9,  (1),  30-38. 
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the  information  available  on  the  radarscope.  By  dividing  fixations  by  the  object  fixated  upon, 
researchers  obtain  the  structure  in  the  visual  scan  between  objects.  This  has  the  potential  to 
reveal  scanning  strategies  or  grouping  of  objects  used  by  ATCSs. 

Researchers  base  another  potentially  useful  division  of  fixations  on  the  distance  between 
subsequent  fixations.  By  creating  bins  based  on  inter-fixation  distances,  one  can  reveal  the 
tightness  of  the  visual  scan.  A  hi^  number  of  transitions  between  or  within  bins  that  represent 
short  distances  could  indicate  closed  loop  control  in  the  visual  system.  Transitions  between  long 
distance  bins  would  indicate  situations  where  local  feedback  cannot  be  used.  This  would 
indicate  higher  level  cognitive  processes  often  thought  to  exist  in  open  loop  control. 

Finally,  one  of  the  goals  of  the  visual  scanning  research  program  is  to  develop  measures  that 
quantify  die  nature  of  visual  scanning  patterns.  Few,  if  any,  studies  have  addressed  a  crucial 
point  necessary  to  develop  such  measures.  Structure  in  a  visual  scan  does  not  reveal  if  the  ATCS 
created  a  situation  that  allowed  efficient  acquisition  of  information  available  on  the  radarscope. 
To  do  so,  one  needs  to  express  the  information  on  the  radarscope  as  a  function  of  time  and 
investigate  if  the  ATCS  picked  up  the  available  information  in  an  efficient  way.  In  other  words, 
visual  scanning  efficiency  creates  a  situation  that  allows  for  maximal  information  pickup. 

The  ATCS  scans  not  only  the  radarscope,  but  flight  strips  and  communication  channels  as  well. 

In  the  current  study,  researchers  recorded  ATCS-pilot  conununications.  Some  ATCSs  are  under 
the  impression  that  ATCSs  conduct  auditory  and  visual  scanning  simultaneously,  i.e.,  while 
looking  at  one  aircraft  an  ATCS  talks  to  the  pilot  of  another.  By  transcribing  the  ATCS-pilot 
communications  and  synchronizing  the  messages  with  the  fixation  information,  verification  of 
this  impression  is  possible.  In  case  aircraft  at  which  ATCSs  looked  strongly  correlate  with 
aircraft  to  which  ATCSs  talked,  processing  is  not  parallel.  If,  on  the  other  hand,  no  correlation 
exists,  this  would  indicate  that  ATCSs  were  talking  to  aircraft  at  which  they  were  not  looking. 
Communications  and  visual  scanning  would  then  happen  in  parallel. 

Knecht,  Smit,  and  Hancock  (1996)  have  used  risk  indices,  calculated  firom  separation 
requirements  and  actual  separation  between  aircraft,  to  look  at  actions  taken  by  pilots  to  prevent 
loss  of  separation.  Similar  indices  can  be  developed  for  ATCSs  and  visual  scanning  variables 
can  tihen  be  compared  to  different  risk  levels. 

The  study  examined  the  differences  in  terms  of  number  of  fixations  and  fixation  durations. 
Researchers  identified  objects  on  the  radarscope  by  type,  e.g.,  aircraft,  airports,  VORs,  etc.  The 
object  group  of  aircraft  can  be  further  broken  down  into  arrivals,  overflights,  and  departtires,  or 
VFR  and  IFR  aircraft.  Fixation  duration  contains  information  about  the  processing  time  that 
provides  msight  into  the  complexity  of  processing  related  to  different  aircraft  types. 

During  the  experiments  and  during  demonstrations  audiences  ask  questions  like  “Can  better  eye 
movements  be  taught?”  A  highly  skilled  visual  scan  evolves  from  years  of  experience.  Another 
approach  taken  by  researchers,  called  the  “optimal  controller,”  states  that  the  “optimal  controller” 
samples  displays  economically  without  compromising  risk  issues.  For  example,  when  two 
aircraft  close  in  on  one  another  one  would  need  to  sample  more  often  when  the  aircraft  grows 
closer.  If,  giving  sampling  of  these  two  aircraft  too  much  priority,  the  risk  of  conflicts  occurring 
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between  otiber  aircraft  not  sampled  increases.  The  optimal  controller  would  sample  optimally. 
Then  researchers  are  able  to  compare  visual  scanning  information  recorded  from  ATCSs  with  the 
performance  of  a  non-existing  optimal  controller.  Research  in  diis  area  frequently  requires  the 
use  of  an  oculometer  to  understand  differences  between  optimal  and  operational  control.  In  an 
operational  setting  one  would  target  adaptive  support  systems  based  on  what  ATCSs  are  most 
likely  to  miss  compared  to  an  optimal  ATCS  model. 


Appendix  N 
Recommendations 

1 .  Modify  the  data  reduction  module  to  incorporate  both  VFR  and  IFR  rules. 

Rationale:  The  data  reduction  module  at  the  RDHFL  does  not  distinguish  between  VFR  and 
IFR  aircraft. 

2.  Modify  the  data  reduction  module  to  calculate  ATCS  performance  based  on  ATCS 
responsibility,  not  on  position  symbol  only. 

Rationale:  Currently  the  data  reduction  module  assumes  aircraft  carrying  a  position  symbol 
belongs  to  a  particular  ATCS.  Aircraft  carrying  the  ATCS  position  symbol  as  well  as  other 
aircraft  inside  the  ATCS  airspace  are  the  ATCS  responsibility. 

3.  Investigate  the  effect  of  an  intrusion  alert,  warning  the  ATCS  of  aircraft  entering  Class  C 
airspace. 

Rationale:  Verbal  reports  during  this  experiment  on  aircraft  intrusions  into  Class  C  airspace 
indicated  that  ATCSs  did  miss  some  of  the  intrusions  .  Their  eye  movement  characteristics 
did  not  change  during  the  5-mmute  intervals  that  included  these  events.  The  features  of  the 
representation  of  these  aircraft  did  not  differ  from  aircraft  under  normal  control. 

4.  Investigate  the  efficiency  of  ATCS  visual  information  acquisition. 

Rationale:  Increasing  task  load  and  introducing  visual  noise  affected  eye  movement 
characteristics  as  evident  from  scene  plane  and  radarscope  object  data.  Eye  movement 
characteristics  by  themselves  do  not  provide  insight  into  how  ATCSs  acquire  information. 

5.  Investigate  how  the  ATCS  uses  fixation  time  on  aircraft  representations. 

Rationale:  The  data  indicated  that  ATCSs  spend  the  most  fixation  time  on  aircraft 
representations.  The  question  remains  as  to  how  the  ATCS  uses  this  time.  The  aircraft 
representation  (radar  return,  vector  line,  and  data  block)  contains  more  information  than  any 
of  the  other  objects.  Does  the  ATCS  spend  more  time  acquiring  this  information,  or  is  the 
increase  in  fixation  time  due  to  an  increase  in  higher  level  cognitive  processing? 

6.  Investigate  if  ATCSs  acquire  all  aircraft  information  during  a  single  fixation. 

Rationale:  One  assumption  is  eye  movements  force  a  sequential  acquisition  of  information. 
With  an  increase  in  expertise,  ATCSs  develop  high  levels  of  automation  in  the  ^quisition  of 
visual  information.  How  much  information  ATCS  can  acquire  during  one  fixation  remains 
unknown. 

7.  Investigate  ATCS  visual  information  processing  in  the  parafoveal  and  the  peripheral  field  of 

view.  .  «•  L 

Rationale:  Some  researchers  have  shown  that  cognitive  load  and  expenence  affects  the 

amormt  of  information  collected  from  a  fixation.  If  experience  increases  the  functional  field 
of  view,  how  much  of  the  radarscope  can  the  ATCS  process  in  a  single  fixation? 
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